Rinie Kervel
Posts: 26
Nickname: rinie
Registered: Oct, 2005
|
|
Re: Software Development Has Stalled
|
Posted: Feb 9, 2010 11:50 PM
|
|
> > not a collection of cell-objects, a sql row is not a
> > collection of column objects.
>
> What alternative are you proposing? I think it's going to
> be difficult for anyone to recreate the at of programming
> in terms of what should not be.
I am still playing around to find alternatives.
In JSON you can look at the data and see how you can remove the overhead of repeating rows by splitting the data in a header/rows construction.
E.g. dbslayer (http://code.nytimes.com/projects/dbslayer) does:
{"RESULT" : {"TYPES" : ["MYSQL_TYPE_LONG" , "MYSQL_TYPE_STRING" , "MYSQL_TYPE_STRING" , "MYSQL_TYPE_STRING" , "MYSQL_TYPE_LONG"] ,
"HEADER" : ["ID" , "Name" , "CountryCode" , "District" , "Population"] ,
"ROWS" : [[3793 , "New York" , "USA" , "New York" , 8008278] ,
[3794 , "Los Angeles" , "USA" , "California" , 3694820] ,
For multiple types of rows (for the tree people, and not strict single type nodes): You can add a level tot the TYPES/HEADER arrays and add a 'nodeKind' index to the rows.
This leads me to examine at least 3 levels of constructs in programs:
- Primitive data and possible procedural code.
- Objects/Components that make sense as individual encapsulated 'things' (GUI/DOM is the way)
- Collections/Dictionaries that manage data. E.g. the dbslayer approach. So you have some kind of dataset where you split the pure data with some header/nodekind manager an nice iterators. The interface to the nodes should be like accessing an individual node object but the storage should not.
An example of the collections approach would be 'strings considered harmfull'. XML/Java like 16 bit characters and strings to consist of those 16 bit character elements.
For loose strings it does not matter which representation you choose: small number of data, so overhead can be ignored. For 1 GB of ASCII or ISO-8859-1 you waste 50%.
Why not store all strings in various dictonaries.
Determine the properties per dictionary and allocate data per dictionary. (Properties e.g. encoding, charactersize, zero terminated, memory, hashed, namespaces/scoping)
This way you build 4 dictionaries (memory, 8, 16 and 32 bit). Choose a default dictionary for loose strings and have efficiency without excluding any encoding.
Of course you can have mixed encodings but searching for ã in an ASCII encoding will result in a hit. So you need some optimisations (convert search string to dictionary encoding versus convert dictornary encoding to search string encoding).
So code needs primitives and collection management besides objects. Data needs to be filtered/sorted/queried before fetching (not foreach but foreSome) and I/O should be async/evented.
|
|
Previous Topic
