Lots of developments and great work here! I have several updates and observations: - On the Mac incompatibility issue: From everything I've seen so far, it appears to be an issue with this specific USB serial chipset (usbmodemA121) and either Mac OS X, or the Java runtime on the Mac. I would need a non-Java terminal program to determine which one of the two it is. I could also install Windows on the iMac, but it's a real pain accessing its hard drive, and I don't want to run it in boot camp on the existing drive. Maybe one day if I have time, I could use one of the old Windows installations from another computer as an external drive. The Digitrax PR3, which uses the same serial chipset, and is probably one of the best tested and most supported JMRI interfaces, also crashes the Mac in the same major way. As you may have guessed by now, TwinDad's driver does the same. Mind you, it's not just JMRI that crashes, it's any serial program that I could try, including Arduino IDE. But last night I tried a new Arduino mini with a different FTDI chipset, and guess what, it works just fine. I'll do a web search to see if anyone reported anything similar. - On the DCC++ driver, before setup: As I was experimenting with different DCC interfaces on the Mac, the last configuration was for the Digitrax PR3, which I had disconnected and plugged in its place the new Arduino mini mentioned above. I had moved the Uno to another room, and I didn't feel like unscrewing the wires from the test track loop. I used the Mac to tested TwinDad's driver, as I didn't want to introduce new variables into my own driver on the Windows computer. The problem now was that JMRI was refusing to change the connection from Digitrax to DCC++. Actually, it was changing it, but it was refusing to save the configuration (null value exception). Another half hour debug session revealed that JMRI was confused that the disconnected (!!!) PR3's serial port wasn't present in the system when attempting to save DCC++ configuration (!?). This is obviously a JMRI bug. A big one. - On the driver: RegisterManager.java: Looking at the source code before I had a chance to run it, I noticed that you use reference counting for keeping track of used registers (slots) and locos, and was wondering what if for whatever reason a throttle object is not properly closed (or disposed), and is left hanging. On my small layout with only four locos that will never be an issue, but what if someone happens to have 50 locos, but runs a few at a time, which I think is fairly common. So I think, this is still okay, they'll restart the layout long before it happens. However, something is not quite right with releasing locos, and the slots don't get reused, so after requesting more than 12 locos (one by one using a single throttle, not all at the same time), no new locos can be used. I'm sure this is an easy fix. - On turnouts: This post is getting to long already, so I'll address this separately. Coming soon...
As I mentioned in another post a few days ago, there is no need to save turnout states on the base station. If you need to cache them, do so in the driver (save to a file), not in the base station. The main issue is that we need to use JRMI turnout objects with accessory commands (<a ...>) rather than with the predefined turnout command (<T...>). The JMRI light objects don't solve the issue, because you can't define routes with them (or am I mistaken?). I depend on routes on my layout, so I find this very important to implement correctly. The predefined turnouts make perfect sense with Gregg's GUI, but in JMRI that function is served by the turnout objects. Let's see how it works: You can throw a turnout by sending a DCC command from a throttle, a signaling system, or by software (as in scripting). You can also throw it manually (think Peco with snap switches), or by pushing fascia buttons. On some systems, like Digitrax Loconet, JMRI is just one of the possible upstream components, along with handheld throttles, signaling systems, and other stuff that communicates with the command station before DCC is sent to the rails. In DCC++, however, you can only have a single upstream component at a time. Gregg's GUI, or JMRI, or something else, but one at a time. Turnouts are controlled by sending a DCC packet to the rails. If we do it any other way than manually, JMRI will know about it, and will tell the driver, which will send the accessory command to the base station, which in turn will put the corresponding DCC packet on the rails. In that specific order. Every time. JMRI caches it during the session. You can save it, so you know the state when the system start next time, if you like. If, on the other hand, someone throws a turnout manually, the system will not know about it, unless we put a sensor. Which will then be reported to JMRI using the sensor driver (which can, of course, be our own DCC++ driver), which can then be used to update the turnout object status. If a turnout is thrown by a fascia button, then again, only the accessory stationary decoder is aware of it, and can't communicate it back to the base station through DCC, so we have the same issue as before. Remember, the DCC rail protocol is one-way only on the main track. So, in both cases, we either use sensors to communicate the actual state of turnouts, or we just always send turnout commands. I prefer the second approach on my own layout, but I can see that someone could prefer the first. Predefined turnouts in the base station are not required for either. Again, this is great work, keep it coming.
A few quick points in reply... (1) I haven't tried switching interfaces as you did between the PR3 and the Arduino. That does seem to be an interesting problem. (2) Indeed, in looking at the WiThrotttle interface (*totally* do-able, by the way), I believe the whole way DCC++ handles the registers may need some re-work, for much the reason you state. There is no good way to handle the release and re-allocation of Registers as time goes on and different locomotives are used. As long as you never (between power resets) use more than 12 locos, what is there will work, but once you need to add that 13th loco, it appears to break. (3) The use of JMRI Lights to access the Stationary Decoder command is just a "hack" as a temporary thing while I dig deeper into Turnout control. In JMRI, pretty much all output devices are "Turnouts". What I currently plan to do is to provide two different feedback methods for the JMRI Turnouts... the "no feedback" version will send a Stationary Decoder command, and the "Direct Feedback" will use the Turnout command. This most closely maps to the way the other systems model things. I think I agree that caching the turnout state on the Arduino isn't necessary, unless the Arduino is running in self-contained automated fashion. It would be VERY helpful within the JMRI system if the Stationary Decoder command could at least have an ACK or OK response, just so the response itself could trigger some things within the program. It doesn't have to carry any information other than "Message Received". Oh, as for the "only one upstream source at a time" ... assuming it works, I've just broken that. I have a "DCC++ Over TCP" interface written for the Arduino code that uses an Ethernet Shield. If it works, one should be able to connect a local JMRI (or Processing or whatever) over the USB, and also hook up a remote JMRI (or whatever, using the protocol) over TCP. And when I get the WiThrottle code working, a possibly arbitrary number of smart phone throttles could be feeding data into the system as well. It's still in my sandbox, but it looks promising. The basic TCP link in particular was surprisingly easy*, since it's just wrapping the serial commands into a TCP packet, and most of the Ethernet link stuff (including DHCP!) is well packaged by the libraries. * ... which means it's probably horribly broken
Let me also take a moment to explain my use of reference counting in the RegisterManager. My assumption is that effectively each of the 12 Registers on the Arduino will be assigned to a particular locomotive address. There could be (in theory at least), multiple devices sending commands to that address. The reference counts keep track of how many devices are using the Register, and mark it as "free" when the reference count is zero. It's my way of working around not being able to free and re-allocate registers within the Base Station, on the assumption that if I send a Throttle command with a different Cab value to a specific Register, I have effectively reassigned that Register to the new Cab. In practice, there should really be only one object (Throttle) accessing each Loco, and thus using each Register. But allowing and managing multiple devices avoids (or at least softens) the need for error handling around trying to grab an already-assigned address.
The registers within the Base Station are nothing more than places to store an arbitrary throttle command that will be sent to the tracks in sequence. For example, you can store 3 throttle commands for the same cab in 3 different registers. Not sure why you would want to, but I've done it accidentally and it's neat seeing the train change its throttle from one speed to the other in rapid succession. Probably not a good idea though. Instead, the idea is to store throttles for different cabs in different slots. These can be changed to any other throttle command at any time, and since throttle commands include the cab address, that means any register can be updated with a new throttle command for a different cab. There is no "releasing" of a register. Just overwrite as needed. The max of 12 is quite arbitrary and is only dictated by memory limitations. You can probably increase it to 20 on the Uno without running out of memory (I'll double check this). On the Arduino Mega, you probably can set this to well over 100 since the mega has a lot more SRAM. In terms of my own next steps, since there is so much amazing work being done on the interface side, I will continue to focus on the Base Station. My plan was to release the version of the code for the Mega that I am using in production on my own layout but I've decided to instead to try to augment the Uno code with optional libraries for the Mega so that one code base can be used for both types of Arduino. In particular, I have a nice RGB driver that includes code to convert HSV colors to RGB. I also have a timer function that is very useful for embedding automation routines in the Mega. Also, I realized that requiring code changes to add and change turnouts or sensors is less than optimal. I am changing this so that turnouts and sensors can be added and edited directly through the command set, including a new command that will store these settings in EEPROM so they only need to be set once. For the turnouts, I understand that JMRI can store the status internally but since others may use alternative interfaces I will keep that function in the the Base Station as an option. Please let me know if there are other extensions or changes to the Base Station that you think would help the interface development.
OK, I think I see what you're saying about the Registers... but the higher level code still needs to choose a register number to supply with the <T...> command... Is it OK, then, for the higher level code to make its own assignment of (for example) Throttle to Register, and then just use that arbitrarily? So for example, for a given throttle to always send commands to Register 3? If so, what if two different throttles controlling two different locomotives choose the same register? This isn't so much of an issue if all the throttles are coming through the same interface (e.g. all JMRI throttles, I can manage throttle<->Register assignments internally), but if multiple interfaces are up, there could be some cross-contamination. For example, when I get the WiThrottle support working, User A might connect directly to the Base Station with a WiThrottle app. The WiThrottle interface might choose Register 3 to write its commands to. User B might later connect a JMRI software throttle directly over the USB port, and JMRI might also select Register 3 to write to. This isn't a problem in terms of sending different commands to the same loco, but if the two throttles are relying on the Base Station to send the periodic refresh packets, whichever throttle sent a command last will get its loco serviced, while the first throttle's command will get lost until a new command is sent. I suppose each higher level interface could issue a Status message to poll the Registers and see what is available before assigning a Register to its throttle, but there is still a (possibly unlikely) race condition while waiting for the status messages to come back and grabbing a new one. In the WiThrottle code I'm writing, I "solved" this by doing the following: (a) Add explicit storage of the loco address to the Register structure (so I can look it up later) (b) Add a getRegisterByAddress() method that takes an Address and returns the Register number that is using that address at the moment (c) Add a getEmptyRegister() method that returns the first unused register in the list (regmap == NULL). (d) Add a freeRegister(int i) method that basically sets regMap = NULL; This is still all just "on paper" since I don't have the hardware to test it yet (it's on order). The idea is, though, that when a WiThrottle client connects, the code can select an unused register or a register already sending commands to that same loco and use that register for its messages. When the client disconnects, the register can be marked as unused and can be used by a different client.
TwinDad, with TCP connections and WiThrottle implemented in the base station I agree that caching turnout states moves to the base station. There are still three layers: interface, command station, and DCC encoding, and turnout caching still belongs in the middle one, it's just that we are putting the last two layers in the same device. Which is also how Gregg has designed it to work with his GUI. Gregg, if I remember correctly, on Digitrax systems slot assignment is done in the base station. A throttle sends a cab address, and the base station replies with the assigned slot number. It also keeps track of in-use slots, and the slot table can be queried. When done with an engine, the user releases or dispatches an address by pressing a button on the throttle, and the system marks the slot as free.
This is close enough to the actual algorithm that it would suit our purpose, should we choose to go this route. I don't think it would be very hard to code up, either. I've already added storing the address in the Register in my branch. A simple integer bit field could keep track of whether each Register is assigned or not. We'd need a reply that indicates "No Register Available", and we'd need commands for requesting and freeing a register. That's about it. Oh, FTR, the way I did the Ethernet (DCC++OverTCP) and WiThrottle support is an interface layer that parses the incoming data stream and then calls SerialCommand:arse(). The Ethernet side is easy, since the actual data is identical to the serial data... just strip off the preamble and trailer flags and forward the "guts". The WiThrottle layer has to do a fair amount of translation between the two protocols, but it's still fairly straightforward. The main thing is that the WiThrottle has to keep track of the different clients (phones) and their throttle settings, because (for example) it has separate Speed and Direction commands, which must be merged into our combined <T> command. I'm really hoping to get hold of an Ethernet shield soon so I can get basic debugging done and share this with you.
In my GUI throttle I can control two locos at one time and each time I add a throttle I can control 2 more locomotives at one time....all this information is "saved" in a hash array and also to a flat file....and each time a command is sent....to speed up or slow down, bell whistle etc it parses the file and grabs the locomotive address. My command code for the throttle is this: <t $leftthrottle $locoaddress 10 1> and each click of the GUI throttle button increases or decreases the speeds. The same holds true for reverse.
So what happens when you add the seventh throttle? Can you delete throttle #3 and then start a new throttle that uses its registers? What if you have two different instances of the GUI running and they can't see how many (or which) throttles the other one has active?
As it stands the DCC++ base station can only handle 12 locomotive slots..or depending on the memory size..I haven't started load testing yet as I am finishing up the GUI...and that's what I am working off of. Each GUI keeps track of 2 locomotive addresses...and if one is DISP ( dispatched off ) that clears that slot for another to be added. As far as the GUI is concerned.....when you add Loco....again based on the digitrax system...it remembers which throttle that address is assigned too....and in this case the throttle will remember which address it has in its memory...and if another GUI tries to take that address the word Steal will pop up...just like it does on Digitrax Forgot to add....each time a locomotive address is added it is given a throttle identifier code that corresponds to each address
Quick question... I got a standard motor shield... do I need to make any modifications, or just plug it into the Mega and hook up the track wires? Also, what about powering the motor shield? Can I hook up a 12V supply to the Mega, or does the motor shield need separate power? And a bit of news... I also got an Ethernet shield... after a few modifications, I have the Ethernet connection (DCC++ over TCP) working, with a static IP and with DHCP. Still having a bit of trouble getting Bonjour/Zeroconf working, but we'll get there. And obviously, with the TCP stack in there, the sketch is too big for an Uno, but there's plenty of room on the Mega. I'm going to shift tacks a bit and hook up the sensors and stationary decoders within JMRI now. I'm not sure when the cutoff for the next release is, but I want to make sure that I make it
I have a Pololu motor shield which I needed to operate HO Scale trains....which works nicely with my 18 volt digitrax power supply....the same power supply that powers my super chief system....for NScale Gregg used a 15 volt power supply directly to the motor shield...do NOT power the Arduino with the power supply...otherwise it will fry the Arduino....and it looks like I will have to either make a homemade Mega or get the Arduino Megashield in order to to take advantage of the modeifications. However I've already done the sensors and signals using the UNO....I will just shift all that to my other UNO sitting here
It's not over the limit by a huge amount. It might be possible to tweak the build and get it in under the wire. I just don't have time right now to try and optimize it. NEW QUESTION: I'm implementing the Sensor (<Q...>) message in JMRI. I notice that in the BaseStation code, the Q message is only sent when the sensor goes Active, and there is no "value" field in the message. I'd like to change this (for JMRI compatibility) to send the message when the state changes either way, and add a boolean "value" field to the message. Old (current) way: <Q Number> ... sent only on inactive->active transition Proposed New way: <Q Number [0,1]> ... sent on both inactive->active and active->inactive transitions. Reason: Sometimes you want to know when it turns off. For example, a block detector. The current implementation has no way of telling JMRI that the block just went empty. I will make this change in my branch, and y'all can decide whether to pick it up officially. Which reiminds me (point #3 this morning) ... Have you thought about having a website to share official documentation and such? I could throw together a Wordpress blog and host it on my site in about ten minutes. It could have links to the YouTube videos, written instructions on putting together a system, documentation such as the official software interface spec (which we're going to need at some point) ...
TwinDad ..... Re: point #3 ..Good idea ..... while I have a working DCC++ set up and I understand the coding you are all doing, you have all gone way above my abilities and this would be a big help. I am anxiously awaiting the next JMRI release (incorporating the DCC++ integration) since the Java, Netbeans, Github method got sort of complicated.
Well, good news on that... the next JMRI release *should* get kicked off this weekend, so your wait won't be long. I just hope that what I get into the release is functional enough I will add the mods to the Sensor class in the Base Station. I will be sending you a giant Pull Request on Git with the Ethernet stuff and related changes soon. One further interface adaptation request... can we add a Query Sensor State command (maybe <q Num>) so that the sensor can be polled? I'll code that up, too
OK, a further update: (1) I fixed the Turnout code... Now if you set a JMRI Turnout to DIRECT feedback, it will use the Accessory Decoder command. If you set it to MONITORING feedback, it will use the Turnout command. In the former case, the JMRI sensor address must match the address of the turnout. In the latter case, the JMRI sensor address must match the predefined turnout number in the Base Station. (2) I added Sensor code. Right now, this requires my change to the Base Station that adds the sensor State to the <Q> reply. I'll be submitting that change to the Base Station code shortly. (3) I tweaked the Simulator so that it will randomly send a Sensor state message every so often. This might have some side-effects. (4) I would have already sent a Pull Request for the Base Station code, but my Mega chose a very inopportune moment to die, so I haven't had a chance to final-test it on hardware yet. Going out at lunch to buy a new one. Also, the next JMRI development release will be 4.1.4, and should be going out in the next few days to a week, fingers crossed.
Um, well... something's got my Arduino hosed up. I have a new one on order but it won't be here for a few days. In the meantime, if you want to examine, and maybe test, the changes I've made, here's my fork of the repository: https://github.com/msunderwd/BaseStation-Uno
All, I've uploaded a new video to the DCC++ channel showing how to use an Arduino Mega 2560 instead of an Arduino Uno. The same Arduino Motor Shield can still be used but the jumper wire configuration is slightly different (see video for details). I have completed (though not yet uploaded) changes to the DCC++ Base Station code to allow dynamic creation/edit/storage of turnout definitions and will be applying the same technique for the sensors. Took me a bit longer than I thought since it required some custom linked-list logic and dynamic memory allocation. But the upshot is that it will avoid the need for users to change any DCC++ code just to add or edit turnouts or sensors. I will also add logic to detect and configure the output pins according to whether an Uno or Mega is being used (and maybe even some logic to dynamically change the number of DCC packet registers). TwinDad - thanks for the extension to the sensor code. I'll take a look and once you've tested it send me a pull request and we can bring it into the main branch. -Gregg
I just updated the Base Station code on GitHub with a changes to the way turnouts are defined. Instead of changing them in the code, you now create them through the interface using <T ID ADDRESS SUBADDRESS>. The <T ID THROW> command still operates the same way. Also, you can delete a defined turnout by specifying only its ID as such: <T ID>. Finally, you can list all turnouts currently defined with a simple <T>. Though turnouts defined dynamically are retained through the session they are not stored in EEPROM until you request storage with the new <E> command. This updates the EEPROM with the definitions of all defined turnouts and also tells the code to keep track of the thrown-status for each of the defined turnouts. The next time the Base Status is restarted, turnout definitions stored in the EEPROM (as well as the thrown-status for each turnout) will be automatically loaded into memory. Also, per some of the suggestions above, these commands return either some data, or an <O> if there is no data to return, but the command is executed OKAY, or an <X> if the command is not executed. For example, if you try to throw a turnout with <T ID THROW>, you'll get an <X> if ID has not yet been defined. Similarly, if you try to delete a turnout using <T ID>, you'll get an <X> if ID is not yet defined. Finally, if you send a <T> command you'll get an <X> if no turnouts have yet been defined. You can safely ignore <X> and <O> returns if you want -- they are just there for some positive feedback if needed. Next up will be a similar change for the definitions of the sensors. Note that to make sure we can keep track of the different versions of the code, I've started issuing release numbers. The original release is tagged as v1.0.0, so you can always restore your local copy to that original version. Once I finish the changes to the sensor code I'll stamp a new release number, such as v1.1.0.
