There have been discussions here and there about alternatives which I hope can be consolidated into one single coordinated effort. One of them assumes that Kalmbach will provide no assistance whatsoever (i.e. we own it, too bad) and the thousands of hours of labor will have to be repeated. Another assumes that Kalmbach will consider donating the entire database to a non-profit organization of some sort which would then take on volunteers to refresh and maintain it. Personally, I think any website or forum that takes this on will jump to pre-eminence among the multiple and growing options that are out there for railroad related information. And I agree that this is a major loss. In preparing the UMTRR most months, I used it at least once to reference my nearly complete accumulation of Rail Model Journal magazines.
Well, Kalmbach came by the Database as a donation. So if they are good people, they should not object to it going out to others as a donation. Personally I think this website would be an excellent choice. MRH's website would also be good, but they are so busy getting their magazine out the door, that I don't think they have the time. Also, I don't think Kalmbach would really have any interest in aiding a competitor. I assume when they say DOS that they mean the Database itself is wrtten in a DOS based Database tool and that it is not a modern Database and so maintaining the interface between it and PHP is non-trivial. Or, perhaps it isn't even using a commercial database tool, but is a Basic program. Obviously converting the information to MySQL is the correct solution. I would not imagine it to be technically challenging. Using a Wiki is a good idea. Installing Mediawiki to do such a thing would be straight forward. It's just a matter of the man years needed to convert the data.
I agree that it came down to money, and not the lack of expertise. The sources are still available to find old Pubs but will have to be searched for. A Shame that most things come down to money now a days.
Whatever database engine it is under, chances are that it's designed around 16-bit installers that simply will not operate under Server 2008 standards. It stops them cold. We just converted in my company and I had several nasty surprises in that area and ended up lifeboating a couple old applications 'off the network' to stand-alone workstations. While the workstations can emulate XP and usually get away with it, installing the database application to the server itself can instantly become a 'no-go' unless you are willing to put it in an application area that has virtually no security and fully sharable directories down to the root level; hence a security risk. It's speculation, of course, but it had a pretty good secondary indexing system in it and if that was DOS, it may have been home-brew or a tweaked application. I've got a client application right now that's stuck in an old version of Foxpro and they can't understand why it simply dies when they upgrade to 2008. Running fine on 2003, well, yeah, a lot of things do! And conversion cost is more than the entire rest of the project upgrade. The unload/reload programs have to be custom-written, and nobody is left that has any documentation on how the original system indexing was designed, we're just guessing by observation. No source code, just executables. And all the index recovery routines run as 16-bit loads too, so all system maintenance has the same problem as the install. It probably can be done, but you need skilled, and likely volunteer, labor to do it.
The Trains.com forum is locking any threads on the subject of the Database. I hope this means they're taking the conversation off line to deal with only the most serious of offers. Again, they didn't do the work to compile this. They got it as a gift. If they don't freely offer it up, I will be very disappointed with them. As for maintaining compatibility. I would imagine that it's simply a matter of getting the data to a willing volunteer who can run the software on an appropriate machine. I would further assume (and maybe I'm wrong) that it would be possible to write a conversion utility to transfer the data. Or do it manually, in either case. I don't see this as being an insurmountable task in the year 2010. Kalmbach just needs to be willing to release the data. Someone in one of the locked Trains.com threads suggested donating it to the NMRA. While they would seem a logical choice, as has been discussed before, they aren't exactly a group that, as a whole, has shown significant MODERN technical competence. I'd like to see it go to a group that does seem to have such knowledge.
Somehow I didn't make the connection on the first time out. After reading your responses I went back and edited my comment to what you see above. Now, I think we are on the same page.
Folks, Without knowing a whole lot more about the code, this is all pretty useless speculation. Let's not disparage Kalmbach without knowing a whole lot more. I'm no conversion genius but I've overseen a ton of them since 1969, and know when I'm ignorant of the technical issues involved. Anyone here who has actual knowledge of how this project was built and maintained over the years is welcome to comment. Anyone else is free to state an opinion, but I see no reason to be mean-spirited toward Kalmbach, or speculate on their motives. That's just ignorance.
As far as I'm concerned, the technical issues are just an aside. They received this boon for free and so I feel it should be freely given to others. I fully realize that the technical issues could be huge and certainly not feasible for them to work on. My concern is purely with how they received the data and therefore the responsibility I feel they have to offer it up to others.
I agree with the philosophy. Yes, the technical issues could be a problem, if the coding reveals details about the site, or vulnerabilities. I suspect they could publish the data in a simple tabular form, and let someone else wrassle with it, but I don't know that. And they might feel they have invested time and effort in wrassling with the original data, and therefore want something back. Or that they own the data, which they probably do legally. That's not unusual today. Much of the software industry was built on essentially free core modules deeloped way back in the 50s and 60s by universities and consortiums. Or given away free (for a while) by AT&T. Now firms own their own modifications of these free cores. But I don't know that either. I think some innovative thinker will find a way to get this project back. I do know that wrassling tabular data into a new database seems easy, but sure isn't in practice. Anyone have an idea of how many entries are in this database? Even a guesstimate? I just dealt with "cleaning" a database of 1000 entries, and it drove me blind and crazy.
Well, easy and time consuming aren't mutually exclusive. Data entry is easy. But it can also be time consuming. I'm thinking the worst case scenario is that you set up the new database or wiki and then you set up a computer with the old one on it and start at the beginning manually copying the data over. Its super simple. But Time consuming as all get out.
I disagree that data entry is easy. And that copying data over is super simple. Firms that charge for these services, even when employing outsourced (i.e., overseas) firms, I think, will agree, given what they charge. Establishing a wiki is easy. Getting committed volunteers to fill it up is super hard in a small specialized area. I think the hardest task will be to dislodge the data from Kalmbach. Has anyone actually talked to them about this? Isn't that the starting point?
On other forums and mailing lists the idea of approaching Kalmbach at the NMRA Convention has been floated. I have not seen anything about any actual discussion or results. If I do learn anything I will pass it along. If there are results from Milwaukee, hopefully they will not be whatever the real-life equivalent of "thread locked" is in the virtual world. Whatever the reason, there could have been more elegant ways of shutting down the index. I get advertising from Model Railroader in my e-mail at least once a week, and they don't seem to have a problem announcing to me months in advance when my subscription is expiring. I don't think it would have been a major effort to send out an advisory that the end of the index was coming. Data entry of this type will be extremely difficult. The "Wayback Machine" version of the page from September 2005 showed "~70490 articles in 6553 magazines, 328 books". Each article has at least one tag and most had more, plus the author cross-reference. And the index had been updated through 2008.
Having been involved in the migration of data (county hospital medical records), something of this nature, and if all data was to be turned over, would take approximately 4~6-months. The setting up and testing of the new database and related interface is another issue (though not a big deal). I would prefer to see this on a Linix server, rather than one operating under Windows, however. Fewer security issues, IMHO.
As a DBA, databases are my bread-and-butter. Data migrations can be EXTREMELY tricky, depending on the source of the data being migrated. Since this index dates back to the days of MS-DOS, it could be built on DBase, Foxpro, Clipper, or even something totally custom written in BASIC, it's impossible to guess. Drivers (software to talk to the database) exist to allow most modern databases and programming languages to get data from DBase, Foxpro, Clipper, etc... Not so for something custom, it may indeed be impossible to get the data out without modifying the custom software. They claim problems keeping the index virus-free, which leads me to think "SQL injection", which would imply one of the "standard" platforms, like DBase or Foxpro, but again, it's impossible to guess. This isn't an impossible task, but it could be an expensive and time-consuming one...
Checking the various discussion mailing lists that I'm on, there was some and some. I read the forward of a letter from Kalmbach stating that they had no plans to make the data available. Later, though, there was some encouraging news. It is being reported that the database may be moving to the NMRA. I don't think it's appropriate for me to be any more definitive than that since I do not have first-hand information. But I'm personally starting to feel a bit more positive that we have not lost this database. Stay tuned... P.S. I used to program in dBase III Plus and Clipper as well...
Being a member of several forums, the coincidence between IT professionals and modelers, and even N scalers, is noticeable. There's a lot of people here that might not know the difference between a turnout and a switch (sorry, had to put that in again!) but I'd bet the whole railroad that the skillset is behind the keyboards here to recover old database technology and move it over. It's just a matter of how it can be allowed. The NMRA could be a good catalyst, and a potential coordinator to involve university resources as well. The 'real paper' maps, drawings, databases and indexes have migrated to university libraries and railroad museums. We have a digital resource here that's just as valuable. The other issue is keeping the index current, and that's not easy either. I don't know how that was being done via Kalmbach.
Ouch! With modern software, that would be small. But if it's in something like dbase, of which I am all too familiar, that is pretty huge. One or multiple tags. Author cross-references. I get a headache thinking about the exceptions a converter will have to deal with. Hey, I loved dbase in the late 80s! But getting data out of it today--my headache just turned into a migraine. Those 16-bit conventions--and many were extended 8-bit conventions--just limited the code. I've received some private messages that efforts are underway to preserve this database. I can probably aid the effort in terms of conversion.
Given the state of the economy I guess Kalmbach are looking at things that cost them money and I can quite understand a thing like this has reached that place where people say "If you want to go there, you need to start from [pointing to the horizon] over there". The cynic in me wonders if unplugging it in this way is partly a means to judge how badly people want it. From there they can decide whether to (and these are just my maybes) hand it to someone else to manage, dump it, or spend some money to get it in shape but then charge for it's use (and I've no problem with the latter by the way).
I resemble that remark... I think that's key... I believe that there will be plenty of volunteers to help get this done, not just in the N Scale arena but elsewhere. The coordination of this work will be paramount. It wasn't being done after 2008 apparently. So there would be a relatively quick need for about two years' worth of data entry, hopefully using the same general protocol in terms of tags and such. If it is in dBase or similar, there might be some relief in some sort of export of the raw information (though I wouldn't know how it's all indexed together). However I know that there are a whole lotta commas in the actual data, including but not limited to the authors' last and first names (e.g. "Kalmbach, Al") so coming up with a delimiter might be an interesting challenge. Back in the previous century when I was playing with this stuff, I recall needing to be, let us say, "creative," at times. I think that if Kalmbach is going to measure the reaction to the pulling of this plug, it might be worth everyone's while to send a brief and courteous message directly to them.
Its not that the NMRA is lacking "MODERN technical competence", as you've stated, its more a lack of people to do the job. Did you know that the NMRA has on staff at the headquarters only 3 paid staffers? One of which is a very "MODERN technical competent" librarian who is doing his best, alone, to get over 10,000 photos cataloged, scanned, and made available to members, researchers, and anyone else who needs the info? Add to that, also trying to get the library itself and all of its donated collections of books, magazines, newspaper articles cataloged and made available. Plus this doesn't count the actual model railroad collections in the Howell Day Museum, that need to be setup for viewing and in many cases, restored? The NMRA, except for those 3 people, is entirely run by volunteers, 99.999% don't live in Chattanooga. Since everyone here seems to be willing to work on the DB, if and when it becomes available to the NMRA, why don't everyone join the NMRA and volunteer your time to get it modernized and back up? Its easy to sit in one part of the country, and make comments about what is lacking about an organization, esp when, you show by your comments, that you know almost nothing about it.
