pb for serializing a big object
1 posts in topic
Flat View  Flat View

Posted By:   Anonymous
Posted On:   Thursday, April 11, 2002 11:02 AM


I have to serialize an object
the main thing in this object is a hastable (String, object)
There are 270.000 entries in that hashtable, the file output weight 25M
to open it on a 350Mhz, it takes 5 minutes

I don't need to open all objects, just some... but deserialization opens all...

how can I select what object to open, using the key (string) ???
I've tried to use ObjectStreamField[] serialPersistentFields but it needs to be final, and I if I declare the array with 270.000 entries, I got some nullexception later...


Re: pb for serializing a big object

Posted By:   Eyal_Lupu  
Posted On:   Saturday, April 20, 2002 03:20 AM


Here is a sketch of a solution. You can manage your own “DB” of serialized objects. All you have to do is to store the serialized objects in a file and also save (serialize) your own indexing schema on the file (in the example I skipped this part). Whenever you need to access an object you have to load the appropriated segment of the file where the objects are stored (using a random access to the file). The first time you will have to look for an object you will need to reconstruct the whole index, but this one is much lighter than loading all of the objects.

Attached a “raw” example I wrote for you, you can improve this example a lot, some ideas:
- You can build your one Hashtable to implement all the process written in the example.
- Some performances issues about the streams

Good luck

eyal lupu

* SerialObjectRandomAccess.java

package serailobjectrandomaccess;

import java.io.*;
import java.util.*;

* @author Eyal Lupu
* @version
public class SerialObjectRandomAccess {
private static String FILENAME="c:\temp\test.dat";
private static int NUM_OB_OBJECTS=4000;

RandomAccessFile mRandomAF = null;

public SerialObjectRandomAccess() {

public void showExample() throws Exception {
// Create the hashtable
Hashtable lDB = new Hashtable(NUM_OB_OBJECTS);
for ( int lIndex=0; lIndex String lKey = "Key_" + lIndex;
ObjOfTypeA lObj = new ObjOfTypeA();
lObj.mStr = "kuku"+lIndex;
lDB.put( lKey, lObj );

// Serial out
Hashtable lOffsets = storeObjects(lDB);//You should save this one to the disk

// Reading in
ObjOfTypeA lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_4");
System.out.println("mStr=" + lTheObj.mStr);

lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_3999");
System.out.println("mStr=" + lTheObj.mStr);

lTheObj = (ObjOfTypeA)loadObject(lOffsets, "Key_424");
System.out.println("mStr=" + lTheObj.mStr);

* @param args the command line arguments
public static void main (String args[]) throws Exception{
new SerialObjectRandomAccess().showExample();

*Stores all objects to file.
*@param iObjects objects to store
*@return index of generated file
private Hashtable storeObjects ( Hashtable iObjects ) throws Exception {
// Prepare the index hashtable, will map object key to offset on file
Hashtable lOffsets = new Hashtable();

// Prepare streams
ByteArrayOutputStream lByteOS = new ByteArrayOutputStream();

long lEndOffsetOfWrittenObject = 0;

// Run over all of the DB
for ( Enumeration lEnum = iObjects.keys(); lEnum.hasMoreElements(); ) {
Object lCurrKey = lEnum.nextElement();
ObjOfTypeA lCurrObj = (ObjOfTypeA)iObjects.get(lCurrKey);

// Into the byte stream
ObjectOutputStream lObjectOS = new ObjectOutputStream(lByteOS);

// Prepare the index entry
IndexEntry lIndexEntry = new IndexEntry();
lIndexEntry.mLength =(int) (lByteOS.size() - lEndOffsetOfWrittenObject);
lIndexEntry.mBeginOffset = lEndOffsetOfWrittenObject;
lOffsets.put(lCurrKey, lIndexEntry);


// Go to the disk
FileOutputStream lFileOS = new FileOutputStream(FILENAME);

return lOffsets;

*Loads an object by key
*@param iIndexes - the indexes ob objects on the file
*@param iKey - the key of the object to look for
*@return the loaded object
Object loadObject ( Hashtable iIndexes, Object iKey ) throws Exception {
// Open the file for random access
if ( mRandomAF == null ) {
mRandomAF = new RandomAccessFile(FILENAME, "r");

// Load the index by the key
IndexEntry lLookupEntry = (IndexEntry)iIndexes.get(iKey);
byte[] lLookupObjectBytes = new byte[lLookupEntry.mLength];

// Read in the object
mRandomAF.read(lLookupObjectBytes, 0, lLookupEntry.mLength);

ByteArrayInputStream lByteIS = new ByteArrayInputStream(lLookupObjectBytes);
ObjectInputStream lObjectIS = new ObjectInputStream(lByteIS);
Object lRebuildObj = lObjectIS.readObject();

// Cleanup

return lRebuildObj;



class ObjOfTypeA implements java.io.Serializable {
String mStr;
int mInt ;
float mFloat = 5.5F;

class IndexEntry implements Serializable {
long mBeginOffset;
int mLength;
About | Sitemap | Contact