pb for serializing a big object
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Anonymous
Posted On:   Thursday, April 11, 2002 11:02 AM

hello

I have to serialize an object
the main thing in this object is a hastable (String, object)
There are 270.000 entries in that hashtable, the file output weight 25M
to open it on a 350Mhz, it takes 5 minutes

I don't need to open all objects, just some... but deserialization opens all...

how can I select what object to open, using the key (string) ???
I've tried to use ObjectStreamField[] serialPersistentFields but it needs to be final, and I if I declare the array with 270.000 entries, I got some nullexception later...

thanx

Re: pb for serializing a big object

Posted By:   Eyal_Lupu  
Posted On:   Saturday, April 20, 2002 03:20 AM

Hi,

Here is a sketch of a solution. You can manage your own “DB” of serialized objects. All you have to do is to store the serialized objects in a file and also save (serialize) your own indexing schema on the file (in the example I skipped this part). Whenever you need to access an object you have to load the appropriated segment of the file where the objects are stored (using a random access to the file). The first time you will have to look for an object you will need to reconstruct the whole index, but this one is much lighter than loading all of the objects.



Attached a “raw” example I wrote for you, you can improve this example a lot, some ideas:
- You can build your one Hashtable to implement all the process written in the example.
- Some performances issues about the streams




Good luck



eyal lupu


/*
* SerialObjectRandomAccess.java
*
*/

package serailobjectrandomaccess;

import java.io.*;
import java.util.*;

/**
*
* @author Eyal Lupu
* @version
*/
public class SerialObjectRandomAccess {
private static String FILENAME="c:\temp\test.dat";
private static int NUM_OB_OBJECTS=4000;

RandomAccessFile mRandomAF = null;

public SerialObjectRandomAccess() {
}

public void showExample() throws Exception {
//
// Create the hashtable
//
Hashtable lDB = new Hashtable(NUM_OB_OBJECTS);
for ( int lIndex=0; lIndex String lKey = "Key_" + lIndex;
ObjOfTypeA lObj = new ObjOfTypeA();
lObj.mStr = "kuku"+lIndex;
lDB.put( lKey, lObj );
}

//----------------------------------------------------------------------
// Serial out
//----------------------------------------------------------------------
Hashtable lOffsets = storeObjects(lDB);//You should save this one to the disk


//------------------------------------------------------------------------------
// Reading in
//-------------------------------------------------------------------------------
ObjOfTypeA lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_4");
System.out.println("mStr=" + lTheObj.mStr);

lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_3999");
System.out.println("mStr=" + lTheObj.mStr);

lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_424");
System.out.println("mStr=" + lTheObj.mStr);
}

/**
* @param args the command line arguments
*/
public static void main (String args[]) throws Exception{
new SerialObjectRandomAccess().showExample();
}

/**
*Stores all objects to file.
*@param iObjects objects to store
*@return index of generated file
**/
private Hashtable storeObjects ( Hashtable iObjects ) throws Exception {
//
// Prepare the index hashtable, will map object key to offset on file
//
Hashtable lOffsets = new Hashtable();

//
// Prepare streams
//
ByteArrayOutputStream lByteOS = new ByteArrayOutputStream();

long lEndOffsetOfWrittenObject = 0;

//
// Run over all of the DB
//
for ( Enumeration lEnum = iObjects.keys(); lEnum.hasMoreElements(); ) {
Object lCurrKey = lEnum.nextElement();
ObjOfTypeA lCurrObj = (ObjOfTypeA)iObjects.get(lCurrKey);

//
// Into the byte stream
//
ObjectOutputStream lObjectOS = new ObjectOutputStream(lByteOS);
lObjectOS.writeObject(lCurrObj);

//
// Prepare the index entry
//
IndexEntry lIndexEntry = new IndexEntry();
lIndexEntry.mLength =(int) (lByteOS.size() - lEndOffsetOfWrittenObject);
lIndexEntry.mBeginOffset = lEndOffsetOfWrittenObject;
lOffsets.put(lCurrKey, lIndexEntry);

lEndOffsetOfWrittenObject+=lIndexEntry.mLength;
lObjectOS.close();
}

//
// Go to the disk
//
FileOutputStream lFileOS = new FileOutputStream(FILENAME);
lFileOS.write(lByteOS.toByteArray());
lFileOS.close();;

return lOffsets;
}

/**
*Loads an object by key
*@param iIndexes - the indexes ob objects on the file
*@param iKey - the key of the object to look for
*@return the loaded object
*/
Object loadObject ( Hashtable iIndexes, Object iKey ) throws Exception {
//
// Open the file for random access
//
if ( mRandomAF == null ) {
mRandomAF = new RandomAccessFile(FILENAME, "r");
}

//
// Load the index by the key
//
IndexEntry lLookupEntry = (IndexEntry)iIndexes.get(iKey);
byte[] lLookupObjectBytes = new byte[lLookupEntry.mLength];

//
// Read in the object
//
mRandomAF.seek(lLookupEntry.mBeginOffset);
mRandomAF.read(lLookupObjectBytes, 0, lLookupEntry.mLength);

ByteArrayInputStream lByteIS = new ByteArrayInputStream(lLookupObjectBytes);
ObjectInputStream lObjectIS = new ObjectInputStream(lByteIS);
Object lRebuildObj = lObjectIS.readObject();

//
// Cleanup
//
lObjectIS.close();
lByteIS.close();

return lRebuildObj;

}


}

class ObjOfTypeA implements java.io.Serializable {
String mStr;
int mInt ;
float mFloat = 5.5F;
}

class IndexEntry implements Serializable {
long mBeginOffset;
int mLength;
}
About | Sitemap | Contact