tag:blogger.com,1999:blog-34067462785429652352024-03-13T14:47:13.221+01:00Technology blogKoen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.comBlogger24125tag:blogger.com,1999:blog-3406746278542965235.post-47824603231370786832017-07-29T00:34:00.001+02:002017-07-29T11:24:37.193+02:00Home automation: heating with ebus and Bulex<div style="text-align: justify;">
One of the targets of my house automation was to integrate heating. Having some familiarity with heating systems, it was already clear that interfering with the internal regulation would be a big no-no. Ideally the heating system needs to offer some kind of open interface to accept sensor values normally sent by a thermostat, but now from the house automation components instead.
The requirements:<br />
<ul>
<li>retrieve basic information from the system. Is it active? Are there any errors?</li>
<li>replace the wall thermostat with house automation components and software</li>
<li>control the temperature of every room separately (where there is need to at least)</li>
</ul>
So the journey started to boldly go where no man has gone before; asking the heating guy if he could deliver a module that offers some kind of "API" to do this. As it turned out, we fell out of warp pretty fast. I will really try to limit my rant here, but heating manufacturers live in another time-space continuum. Where most software companies need to reinvent themselves every 10 years or so, heating manufactures can do what they want so it seems.<br />
Instead of offering a somewhat flexible interface, most of them are starting to sell "Internet thermostats" to end users. Which is great if you want to retrofit but not if you want to integrate in a real house automation system. Of course, those modules can probably be 'hacked' in the sense that if a mobile app can operate it, it must have some kind of interface. However, most of these modules still depend on a fixed thermostat in the house for transmitting the actual room temperature. So it will only be a partial solution to the problem.<br />
<br />
To be fair, there are vendors that offer complete integration solutions. Buderus for example offers a KNX module for this, but these are ranged for "professional customers" for integration in building control. They are expensive, require you to have an even more expensive control unit (only those units allow to connect the KNX module) and it appears that in the case of Buderus using the KNX module means that all regulation has to be done via the module as it takes over the internal regulation.<br />
The most realistic option seemed to be Viessmann together with the Vitogate 200 KNX module. This module allows high level operation replacing a thermostat while leaving all other regulation up to the heating system. The problem however was the overall price tag. A Viessmann system would be 3 times more expensive than comparable systems.<br />
<br />
So, in the end I decided to go with Bulex. For the record, Bulex is a well known brand in Belgium. In France they are known as Saunier-duval and in Germany as AWB. These brands are the exact same and all belong to the Vaillant group and offer a good price/quality deal. They don't have all those nifty modules or possibilities you get with brands like Viessmann, hell, don't even expect a decent manual, but quality wise they are more than OK.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<a href="https://1.bp.blogspot.com/-A282Be7fe8Y/WXxGh18IutI/AAAAAAAAA3Y/7LPYhqElpa4iVNjqx2YtH1_LufDORyQygCLcBGAs/s1600/EBus_Logo2.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1345" data-original-width="1600" height="168" src="https://1.bp.blogspot.com/-A282Be7fe8Y/WXxGh18IutI/AAAAAAAAA3Y/7LPYhqElpa4iVNjqx2YtH1_LufDORyQygCLcBGAs/s200/EBus_Logo2.png" width="200" /></a>Fortunately, after some investigation it turned out that not offering any integration modules does not have to be a show stopper per se. Bulex, like other Vaillant products, use Ebus as a communication medium between devices (<a href="https://en.wikipedia.org/wiki/EBUS_%28serial_buses%29" target="_blank">Wikipedia</a>). Ebus is more or less open in the sense that information on how the different layers should work is publicly available: <a href="http://ebus-wiki.org/lib/exe/fetch.php/ebus/spec_prot_12_v1_3_1_e.pdf" target="_blank">Physical and data-Link layer</a> and <a href="http://ebus-wiki.org/lib/exe/fetch.php/ebus/spec_prot_7_v1_6_1_e.pdf" target="_blank">application layer</a>. This information is enough to build software yourself that can send and receive commands. Besides the software one also needs a module that can be physically connected to the ebus. As it turned out, there is already an existing solution for both, back in business we are!<br />
<br />
<a href="https://3.bp.blogspot.com/-Xw6qpfK3ogY/WXZL4BFFI4I/AAAAAAAAAzk/6Sjtleygv1MHXdRZL1adHkuRYj3kl7GhgCPcBGAYYCw/s1600/coupler.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1122" height="200" src="https://3.bp.blogspot.com/-Xw6qpfK3ogY/WXZL4BFFI4I/AAAAAAAAAzk/6Sjtleygv1MHXdRZL1adHkuRYj3kl7GhgCPcBGAYYCw/s200/coupler.jpg" width="140" /></a>The ebus USB adapter (I got mine here: <a href="https://www.eservice-online.de/shop/ebus/135/1-wire-hub-platine?c=20" target="_blank">eservice-online</a>) is something that can be custom build. However, at a
price of 75euro this was not worth it for me.
Some warning statements: there is also a Ethernet version of the ebus
adapter on eservice-online. As I started out with this adapter first, my
experience is that this does not work as expected. The adapter seems to
buffer values that it reads from the bus and sends them with a delay
over the network. While this is not an immediate problem, it makes it
difficult to discover which command does what as there is no instant
relation between doing and action and seeing the command in the
software. While this "buffer" value can be adjusted, I was not able to
get it to work properly. Another problem is the configuration of the module (like ip
address, ...). For that it requires a piece of crappy software (that
only runs on windows) and is rather problematic to work with. my advice:
use the USB adapter. It costs 50% less and the device itself requires
no configuration. In my case plugging it in on a raspberry PI 3 was all
that needed to be done.<br />
<br />
Next, besides having an adapter, one also needs software. Over at Github there is a project called 'Ebusd', John30, the creator, is doing an awesome job building and maintaining this which does everything you want with <a href="https://4.bp.blogspot.com/-XTtGdPEHWHo/WXZNLDCPiiI/AAAAAAAAAzk/_8-PuQ9uzGoD_78Gd3uZpHYiR3j9rzKVwCPcBGAYYCw/s1600/github.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="665" data-original-width="800" height="166" src="https://4.bp.blogspot.com/-XTtGdPEHWHo/WXZNLDCPiiI/AAAAAAAAAzk/_8-PuQ9uzGoD_78Gd3uZpHYiR3j9rzKVwCPcBGAYYCw/s200/github.png" width="200" /></a>ebus (<a href="https://github.com/john30/ebusd/" target="_blank">https://github.com/john30/ebusd/</a>). Without his effort my little project would probably never existed. The software comes with MQTT support and a TCP server so communicating with it is a breeze. So basically you hook up the adapter to a spare ebus connection on the control unit, plug-in the adapter via USB into a device (RPI in my case), launch ebusd et voilà, you are seeing all commands that are being send on the bus. The eventual schema looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-TfYvatiIFu4/WXh8UcGLC4I/AAAAAAAAAzs/0ua3uCHe8D4Gu14d5dNs1xO2Z9maZv3dwCLcBGAs/s1600/s1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="443" data-original-width="1101" height="256" src="https://3.bp.blogspot.com/-TfYvatiIFu4/WXh8UcGLC4I/AAAAAAAAAzs/0ua3uCHe8D4Gu14d5dNs1xO2Z9maZv3dwCLcBGAs/s640/s1.png" width="640" /></a></div>
<br />
Some background: the "Room controllers" are parts of the original Bulex system (in my case bulex exacontrol wireless). They are the wireless thermostats that we are trying to replace. They communicate with the heating controller (in my case Bulex examaster) using some wireless protocol which is not important anyway as our target is to eliminate them. The outside temperature sensor is important as this is a weather depended regulation. As an extra we will also be reading out it's value so we can show he outside temperature without having to buy a standalone temperature unit.<br />
<br />
Now, the tricky part are the commands. While ebus might be a standard, the commands used by heating vendors are not. Ebusd comes with config files for some devices that have already been discovered. Based on the device id (broadcasted on the bus) ebusd will load a matching config file and display the representation of the commands.<br />
<br />
Unfortunately for Bulex there is no corresponding mapping to be found. But, playing around with other Vaillant mappings, the first command I found was the one for the outside temperature: <i>15b50903293c00</i> The address layout can be seen here: <a href="https://github.com/john30/ebusd/wiki/HowTos" target="_blank">https://github.com/john30/ebusd/wiki/HowTos</a> quote:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">"QQZZPBSBNNDD / NNDD = count Here, QQ is the source address, ZZ is the destination address, PBSB is the primary and secondary command byte, NN is the number of data bytes DD following, and count is the number of times, the message was seen on the bus."</span><br />
<br />
<a href="https://3.bp.blogspot.com/-xC4ZL2QIBro/WXUNIo9cU2I/AAAAAAAAAyQ/5kWg0I_XwSkYdwibLyWbMp_j0TzjO7NnQCPcBGAYYCw/s1600/ebusctrltemp.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="382" data-original-width="782" height="195" src="https://3.bp.blogspot.com/-xC4ZL2QIBro/WXUNIo9cU2I/AAAAAAAAAyQ/5kWg0I_XwSkYdwibLyWbMp_j0TzjO7NnQCPcBGAYYCw/s400/ebusctrltemp.png" width="400" /></a>When sending this command using <i>ebusctl</i> (a CLI for ebusd, which you get after installing ebusd) the response is: <i>053c008a0000</i>. Based on a response without pre existing mapping one cannot know which byte is what However, since the outside temperature is fairly standard, I found out in the existing mapping files that the the two bytes 0x8a00 represent the temperature value in data2c datatype.<br />
<br />
<a href="https://4.bp.blogspot.com/-_mUjoxa6sHQ/WXUOjfoTG9I/AAAAAAAAAyQ/kftt7cjSdxc272Jq7aL0IWgGwy6Ea0uwwCPcBGAYYCw/s1600/ebusdata2c.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="364" data-original-width="418" height="277" src="https://4.bp.blogspot.com/-_mUjoxa6sHQ/WXUOjfoTG9I/AAAAAAAAAyQ/kftt7cjSdxc272Jq7aL0IWgGwy6Ea0uwwCPcBGAYYCw/s320/ebusdata2c.png" width="320" /></a>Given the ebus standard we can convert this to decimal as follow: reverse byte order 0x8a00 becomes 0x008a. Convert high byte to decimal and multiple by 16. 0x00 = 0 and 0 * 16 = 0. Convert high nibble of low byte to decimal (0x8 = 8 decimal) and add it to result: 0 + 8 = 8. Convert low nibble of low byte to decimal (0xa = 10) and divide it by 16 (4/16= 0.625) add it to the result: 8 + 0.625 = 8.625°C<br />
<br />
<br />
<br />
<br />
<br />
This can also be verified by looking at the ebusd log file. Since I started ebusd with a vaillant config file, it will try to decode every command it sees. The outside temperature is one of the commands that are apparently a bit universal over the vaillant products and is one of the few which is recognized ootb. The temperature is also broadcasted by the control unit on a regular base (you can actively query for it using the command above, or wait for it to pass by on the bus via the broadcast). So here we see ebusd decoding the value of the temperature broadcast. As can be seen, this matches.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-jv-7VK0tVPA/WXZPjCk0xGI/AAAAAAAAAzo/nPrrJnNcXVQrizZTeju8ok2acIuy4aKNACPcBGAYYCw/s1600/ebustemp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="558" data-original-width="1090" height="324" src="https://2.bp.blogspot.com/-jv-7VK0tVPA/WXZPjCk0xGI/AAAAAAAAAzo/nPrrJnNcXVQrizZTeju8ok2acIuy4aKNACPcBGAYYCw/s640/ebustemp.png" width="640" /></a></div>
<br />
So what we have now is an interface into the heating system. As ebusd runs a TCP server ootb, we can now simply write some code to send the commands and receive the value:<br />
<br />
<pre class="brush: java;">
public List<string> send(EbusCommand ebusCommand) throws Exception {
try (Socket clientSocket = new Socket(getInstance().getEbusdIp(), getInstance().getEbusdPort())) {
List<string> results = new ArrayList<>();
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
for (String command : ebusCommand.getEbusCommands()) {
logger.debug("Writing to ebus: hex " + command);
out.writeBytes("hex " + command + "\n");
String result = in.readLine();
logger.debug(" Result:" + result);
if (!ebusCommand.withResult() && !result.equals("00")) {
logger.error("Command hex " + command + " resulted in " + result + " should have been 00");
}
results.add(result);
in.readLine();
sleep(200);
}
return results;
}
}
</pre>
The code above can be found here (<a href="https://github.com/koen-serneels/HomeAutomation" target="_blank">https://github.com/koen-serneels/HomeAutomation</a>) but the bottom line is it sends "hex " + command, in the above example "<i>hex 15b50903293c00</i>" and it receives "<i>15b50903293c00</i>". The only thing left to do is extract the payload and convert the datatype.
So far so good!<br />
<br />
Unfortunately this is where the warm cosy feeling went cold and dark. The assumption was that when changing the temperature on the wireless thermostat the corresponding command would be visible on the bus. After capturing it it should have been possible to simply replay the commands using ebusd, creating an index of useful commands. Unfortunately this was not working. The most likely explanation was that the wireless thermostat would communicate directly with the control unit but the control unit would be smart enough to not put these command on the wired bus as it is the final receiver.<br />
After giving it some thought I decided to place a small bet and get myself a wired ebus thermostat instead (the normal bulex exacontrol). Making the schema like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-adRO_JcqnBQ/WXUPlDulyRI/AAAAAAAAAyY/vVJPk67mSvQpdaSOg6Mj2L8DllqwePmhwCPcBGAYYCw/s1600/aaa.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="472" data-original-width="978" height="308" src="https://1.bp.blogspot.com/-adRO_JcqnBQ/WXUPlDulyRI/AAAAAAAAAyY/vVJPk67mSvQpdaSOg6Mj2L8DllqwePmhwCPcBGAYYCw/s640/aaa.png" width="640" /></a></div>
<br />
<br />
Luckily this worked. Commands issued by the wired thermostat became visible on the bus. In hindsight, getting a wired thermostat was not really required. What happens is that when the control unit receives a command (no matter the source, wireless or wired) it changes an internal register based on the received the command. So the command basically addresses a register with a given value. Scanning all of the registers before changing the temperature and scanning them again after the change, would also reveal the changed registers without having to intercept the actual command.<br />
This is a bit more hassle as reading all the registers takes some time (~2 minutes) and one also need to filter out the changes done by the system in the mean time (these changes would be triggered by internal housekeeping, time depended stuff, updates from other ebus connected devices such as the heater, ...). To be sure one has the real register related to the command the procedure needs to be repeated a couple of times, diffing the files each time and eliminating the non related ones. Anyway, this technique works and it is also required to find out registers for internal state which are never read or send to/from thermostats in the first place
and works no matter the type of thermostat.<br />
<br />
Below a snippet (<a href="https://github.com/koen-serneels/HomeAutomation" target="_blank">https://github.com/koen-serneels/HomeAutomation</a>) from a small tool that scans the range for 0x29 from 0x00 till 0xFF<br />
<pre class="brush: java;">
public class EbusRegisterReader {
public static void main(String[] args) throws Exception {
String template = "15b5090329%s00";
try (Socket clientSocket = new Socket("192.168.0.10", 8888)) {
List<String> results = new ArrayList<>();
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
for (int i = 0; i < 255; i++) {
String cmd = String.format(template, leftPad(new BigInteger("" + i).toString(16), 2, "0"));
out.writeBytes("hex " + cmd + "\n");
String result = in.readLine();
System.err.println(cmd + " - " + result);
in.readLine();
}
out.close();
in.close();
}
}
}
</pre>
<br />
There seem to be two ranges: 0x29 (as shown in the example) and 0x0D. Both ranges can be read. The bytes following denote the rest of the register. I'm not really sure what the 0x29/0x0D is, it's either an instruction for reading or simply a part of the register. Anyway, scanning both 0x29 and 0x0D from 0x0000 till 0xFFFF should scan most if not all available registers. Note that scanning all of these addresses takes a lof of time, especially since only a small portion of the address can actually be read (the others produce a read error). To fix this one can simply write the addresses that returned something meaningful to a file and use that file the next time instead of re-scanning everything.<br />
<br />
Another important aspect is that the control unit identifies the heating circuit (HC) that needs to be controlled using the base or source address. In my case there a two heating circuits. One for the floor heating at ground floor and one for the floor heating at the first floor. When thermostats are first linked with the control unit, each gets a specific base address which is then later on used to identify the HC to be operated. So, in my case, sending a command with source 0x30 operates HC1, sending that same command with source 0x70 operates HC2. The command in combination with the source address determines which HC will be operated. These source addresses seem to be fixed, at least for my configuration (if there are more heating circuits or other components these address could be different). Thankfully ebusd implemented my change request to support changing source address on the fly (<a href="https://github.com/john30/ebusd/issues/50" target="_blank">https://github.com/john30/ebusd/issues/50</a>) so now it is possible to send along the source address with the command as opposed to the source address only being statically configurable at ebusd startup.<br />
<br />
Note that depending on the HC, registers have sometimes different meaning. So a register that accepts the desired temperature (for example) could be different for HC1 and HC2. To be sure you have to test with both circuits to verify the actual registers to use. In my case this makes sense as HC1 and HC2 are of a different type. I won't go into detail here, but by default the system only allows 1 low temperature circuit or 'mixed circuit' (for floor heating) and 2 where needed, so we came with a workaround by using a 2nd HC but of type 'high temperature', but configure it to not exceed 38°C water temperature making it virtually equal to a low or mixed circuit. While the control unit operates the mixing valve for HC1 (as being a genuine low temperature/mixed circuit) it does not for HC1. This is however not a problem as the heater modulates, so it does not need the mixing valve in the first place. It would be needed if there was an actual high temperature circuit (for traditional radiators for ex.) but this is not the case, so all good!<br />
<br />
At this point we have created an interface into the heating system. With some more playing with said techniques, following commands were found:<br />
<ul>
<li>Get the outside temperature</li>
<li>Get heating demand status of heating circuits 1 and 2</li>
<li>Get the target water temperature</li>
<li>Get heating circuit enabled status</li>
<li>Set the desired temperature</li>
<li>Set the current temperature</li>
<li>Set heating circuit enabled</li>
</ul>
All the commands are available here: <a href="https://github.com/koen-serneels/HomeAutomation/tree/master/rpi-adc-dac-controller/src/main/java/be/error/rpi/ebus/commands" target="_blank">https://github.com/koen-serneels/HomeAutomation/tree/master/rpi-adc-dac-controller/src/main/java/be/error/rpi/ebus/commands</a></div>
<div style="text-align: justify;">
<br />
Things still left to do are read/set the hot water temperature, this will be fairly easy to do. And finally detecting if there is any error in the system. This will probably be harder as I will first have to trigger an error :) but especially the latter is a nice to have and not really mandatory.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="https://1.bp.blogspot.com/-VCwaJUfLRZQ/WXURv2LbpZI/AAAAAAAAAyo/qfDFaY8FK6gNzVDK1j9JeO_aSm5Gu4ZbgCPcBGAYYCw/s1600/stellantrieb.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="1200" data-original-width="1200" height="200" src="https://1.bp.blogspot.com/-VCwaJUfLRZQ/WXURv2LbpZI/AAAAAAAAAyo/qfDFaY8FK6gNzVDK1j9JeO_aSm5Gu4ZbgCPcBGAYYCw/s200/stellantrieb.jpg" width="200" /></a>Finally we need something for independent room temperature control. Whether or not this makes sense depends on the situation and type of house. In my case, the ground floor is a more or less open space, there it would not make sense to control individual floor heating circuits. However, on the first floor there are different bedrooms and a bathroom each having different temperature requirements.
In case you're not familiar with floor heating, it are basically different circuits of tubes connected to a manifold. Normally there is at least one circuit per room (possibly more if the surface requires it). So if a room has lets say 2 circuits, one could control the temperature by opening or closing valves on the manifold. This is exactly what we are going to do. Each valve on the manifold is operated by an electronic valve. In my case it are 230v NC valves (in german: stellantrieb) for example <a href="https://www.theben.de/en/Products/Accessories/Climate/Actuator-ALPHA-5-230-V" target="_blank">https://www.theben.de/en/Products/Accessories/Climate/Actuator-ALPHA-5-230-V</a> </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="https://4.bp.blogspot.com/-g7Us4xRgELs/WXpG1OebgfI/AAAAAAAAA0A/XMaHGnnoUT8bnGgwfTDsVu75Uc8IUjzhQCLcBGAs/s1600/mdt.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="316" data-original-width="291" height="200" src="https://4.bp.blogspot.com/-g7Us4xRgELs/WXpG1OebgfI/AAAAAAAAA0A/XMaHGnnoUT8bnGgwfTDsVu75Uc8IUjzhQCLcBGAs/s200/mdt.png" width="183" /></a>The valves are controlled by an MDT KNX heating actuator (<a href="http://www.mdt.de/EN_Heating_Actuators.html" target="_blank">http://www.mdt.de/EN_Heating_Actuators.html</a>).
The actuator can function in different modes. In automatic mode you pass it the current room temperature and based on the set-point temperature it will in PWM style drive the valve so that it obtains the required temperature. There is also manual mode in which it simply opens or closes the valve. The manual mode it's just acting as an relay, turning the valves on or off. In my setup the actuator is in manual mode. The actual controller is self-written. In this case I could just have used 'simple' relays instead of an actuator, but relays also need to be controlled, space and energy efficient, you need to know which 'position' they are in, durable and so forth. In the end the KNX controller is not that much more expensive and it can still do some important housekeeping stuff that I do not need to program, for example it can operate the valves from time to time in summer mode to make sure they don't get stuck. Ok, thanks to these valves it is now possible to control each room independently. Each room corresponds to at least one valve, some rooms have multiple circuits, in that case the combined vavles for that room will be operated as one.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<a href="https://2.bp.blogspot.com/-gd732qqOQ3Q/WXuzNT_1l_I/AAAAAAAAA1Y/Q7Dg2fFq3h8VGACUlYzxivVb5Jjy_MS3wCLcBGAs/s1600/controller2.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="649" data-original-width="658" height="196" src="https://2.bp.blogspot.com/-gd732qqOQ3Q/WXuzNT_1l_I/AAAAAAAAA1Y/Q7Dg2fFq3h8VGACUlYzxivVb5Jjy_MS3wCLcBGAs/s200/controller2.png" width="200" /></a>The only thing we need to do now is glue everything together. For this I have written a heating controller that gathers the current/required temperatures, communicates them to the control unit via ebus and operates the valves.<br />
The current and required temperatures are obtained from sensor in each room. In my case these are build-in into the KNX wall switches. Even though most rooms have motion sensors, where it makes sense, there is also a switch in every room allowing an elegant way (among others) to do these measurement. Next to setting the desired temperature via the web app, it can also be done on the switches directly.</div>
<div style="text-align: justify;">
<br />
So what basically happens is that the <a href="https://github.com/koen-serneels/HomeAutomation/blob/master/rpi-adc-dac-controller/src/main/java/be/error/rpi/heating/RoomTemperatureCollector.java" target="_blank">room temperature collector</a> receives the current and desired temperature (from the wall switches and/or web app) and sends them to the <a href="https://github.com/koen-serneels/HomeAutomation/blob/master/rpi-adc-dac-controller/src/main/java/be/error/rpi/heating/HeatingController.java" target="_blank">heating controller</a>. Based on this it knows if there is heating demand (=there is at least one room of which the desired temp is greater than the current temp) and which room has the highest heating demand. This software is running on the same RPI as ebusd is running.<br />
<br />
But now comes the trick. The heating system in my case has only 2 circuits. One for ground floor and one for first floor. This means it was originally foreseen for 2 thermostats and 2 circulation pumps. On the first floor there are now 4 rooms that require individual temperature control (and actually 4 thermostats). To solve this the heating controller simply communicates the values of the room with the highest heating demand to the heating control unit via ebus. It basically simulates a thermostat that we (virtually) move around all the time into the room with the highest demand. By doing this the heating circuit will only be enabled when there is at least one room with heating demand. From there on everything is controlled with the valves by the <a href="https://github.com/koen-serneels/HomeAutomation/blob/master/rpi-adc-dac-controller/src/main/java/be/error/rpi/heating/RoomValveController.java" target="_blank">room valve controller</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For example, there are 2 rooms in demand of heating. The data from the room with the highest demand is communicated via ebus. The heater operates HC1. In this case the valves for room 1 and 2 open. The valves for the other room remain closed as they don't have heating demand. Suppose room 1 reaches its desired temperature: the valves for room 1 now close. The heating controller will now communicate the temperatures for room 2, still having heating demand. Finally, if room 2 reaches it s desired temperature, the valves for room 2 close and if no other heating demand exists the heater will turn off HC1. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><i><span style="font-family: inherit;">Note: it is important that the manifold that is being used has a "bypass" that still allows water to circulate if all valves are closed. Since the heater is not in control of the valves (we are with our software), there is no real guarantee in what will happen. For example, in case of heating demand the heater might switch a heating circuit to 'on' will all valves are still closed. Opening a valve goes slowly (~1 - 2 minutes) so there might be a short time in which the circulation pump operates with all valves closed. This is not a problem at all, as long as the water can circulate. </span>Although<span style="font-family: inherit;"> modern pumps normally have all kinds of protection, it's better to be safe than sorry.</span></i></span></div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
Finally some screenshots how the control looks from the webapp (the webapp is currently created via Loxone). On the right, the general status and the temperature control from the bathroom is shown. Currently it's summer and the heating is turned off. In that case all rooms go to"frost protect" mode meaning they are set to 10°C. On the left there is the schedule that is activated from the moment the heating goes out of frost protection<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-b_g-ndGTpNI/WXu7uBUA5gI/AAAAAAAAA1w/zupVaVOmA8QJZI-CF-MJ48BUsn4ROFSXwCLcBGAs/s1600/schedule.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="657" data-original-width="1330" height="316" src="https://4.bp.blogspot.com/-b_g-ndGTpNI/WXu7uBUA5gI/AAAAAAAAA1w/zupVaVOmA8QJZI-CF-MJ48BUsn4ROFSXwCLcBGAs/s640/schedule.png" width="640" /></a></div>
<div style="text-align: justify;">
<br />
<br />
In the end I'm happy with the result. The system is running stable for over a year now and nearly all requirements were fulfilled, more than I had bargained for to be honest. However, was this worth all the energy? Yes, as long as you like taming heating systems. Should you do this? Probably not. Get yourself a Viessmann with KNX module and live long and prosper :-)</div>
Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com7tag:blogger.com,1999:blog-3406746278542965235.post-59869250234102347002016-01-30T13:34:00.003+01:002018-07-05T08:20:03.002+02:00PoLA and HttpURLConnection<div style="text-align: justify;">
If you are a developer like me you probably heard of '<a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">PoLA</a>' or the Principle of Least Astonishment. The principle is fairly simple. Never surprise a user. Or in this case even more important, never surprise a developer. Unfortunately I got more then I bargained for last week when one our web service clients was generating rogue requests.</div>
<br />
<div style="text-align: justify;">
Rogue requests you say? Yes, like in; we have no freaking idea where they're coming from. It's one of those moments where managers start running around in circles, panicking and yelling "we're hacked!!" or "please someone turn off Skynet!!".</div>
<br />
<div style="text-align: justify;">
Anyway, first a little bit of background. In our project we have automatic activity logging which is triggered by aspects when a process starts. This includes the web service client in question and the processing on the endpoint as both of them are part of our system. So at some point we noticed that before the response was sent by the endpoint, a second call from the same web service client came in. This was unexpected as the client is single threaded and no other clients were in the picture. Review and tests pointed out that it was simplyimpossible©® for our client to simultaneous generate another request while the first one was still in progress.</div>
<br />
<div style="text-align: justify;">
After a long day of debugging and going through too many logs it turned out that the client was in fact disconnected before processing ended on the endpoint. So there requests weren't simultaneous after all. But why did that took us a day to find out? Did we play starcraft2 again instead of working?</div>
<br />
<div style="text-align: justify;">
Well no, it all started with the HTTP read timeout on the endpoint's container being unexpectedly set lower than we thought. The logging on the endpoint indicated that a reply was generated but the client was actually disconnected before that event because of the read timeout. This was of course not logged by our aspects on the endpoint side as this is decided on a lower level (the HTTP stack) rather then our endpoint itself.</div>
<div style="text-align: justify;">
Ok, true, I hear you say, but what about the web service client log? The web service client should have thrown a "ReadTimeoutException" or something similar and that should have been written to the log, right? Well, true, but it didn't. And now it comes, as it turned out the real surprises is inside HttpURLConnection (more specifically the default Oracle internal handler sun.net.www.protocol.http.HttpURLConnection)</div>
<div style="text-align: justify;">
Did you know that this default impl of HttpURLConnection has a special "feature" which does HTTP retries in "certain situations"? Yes? No? Well, I for once didn't. So what happened was that the timeout exception was indeed triggered on the web service client but silently catched by HttpURLConnection itself by which it decided to do an internal retry on its own. This means that the read() method called the web service on HttpURLConnection remains blocked, like you are still waiting for the response of the first request. But internally HttpURLConnection is retrying the request more then once and thus generating multiple connections. This explained why it took us so long to discover this as the second call was never logged by our code as it is in fact never triggered by our code but by HttpURLConnection internally.</div>
Here some code illustrating this:
<br />
<pre class="brush: java;">import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.SocketTimeoutException;
import java.net.URL;
import java.util.concurrent.Executors;
import com.sun.net.httpserver.HttpServer;
/**
* Created by koen on 30/01/16.
*/
public class TestMe {
public static void main(String[] args) throws Exception {
startHttpd();
HttpURLConnection httpURLConnection = (HttpURLConnection) new URL("http://localhost:8080/").openConnection();
if (!(httpURLConnection instanceof sun.net.www.protocol.http.HttpURLConnection)) {
throw new IllegalStateException("Well it should really be sun.net.www.protocol.http.HttpURLConnection. "
+ "Check if no library registered it's impl using URL.setURLStreamHandlerFactory()");
}
httpURLConnection.setRequestMethod("POST");
httpURLConnection.connect();
System.out.println("Reading from stream...");
httpURLConnection.getInputStream().read();
System.out.println("Done");
}
public static void startHttpd() throws Exception {
InetSocketAddress addr = new InetSocketAddress(8080);
HttpServer server = HttpServer.create(addr, 0);
server.createContext("/", httpExchange -> {
System.out.println("------> Httpd got request. Request method was:" + httpExchange.getRequestMethod() + " Throwing timeout exception");
if (true) {
throw new SocketTimeoutException();
}
});
server.setExecutor(Executors.newCachedThreadPool());
server.start();
System.out.println("Open for business.");
}
}
</pre>
If you run this, you'll get:
<br />
<pre>Open for business.
Reading from stream...
------> Httpd got request. Request method was:POST Throwing timeout exception
------> Httpd got request. Request method was:POST Throwing timeout exception
Exception in thread "main" java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
...
</pre>
Notice that our httpd got two calls while we only did one? If we re-run this, but this time set the magic property <b><a href="http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html">-Dsun.net.http.retryPost=false</a></b> we get:
<br />
<pre>------> Httpd got request. Request method was:POST Throwing timeout exception
Exception in thread "main" java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
...
</pre>
<div style="text-align: justify;">
Putting all this aside, who the hell builds a retry mechanism that isn't really documented nor configurable? Why am I after 15years of Java development (and a network fetish) not aware of this feature? But more over, why the hell is it doing retries on a freaking HTTP freaking POST? PoLA breach detected!</div>
<div style="text-align: justify;">
As you probably guessed by now it's a bug (<a href="http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6382788">http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6382788</a>). Not the retry mechanism of course, that's just crap. The bug is that it also happens for a POST (which is by default not idem potent per HTTP RFC). But don't worry, Bill fixed that bug a long time ago. Bill fixed it by introducing a toggle. Bill learned about backward compatibility. Bill decided it's better to leave the toggle 'on' by default, because that would make it bug-backward-compatible. Bill smiles. He can already see the faces of surprised developers around the globe running into this. Please don't be like Bill?</div>
<div style="text-align: justify;">
So after some exciting days of debugging the solution was kinda lame. Merely setting the property to false fixed it. Anyway it surprised me enough to write a blog entry on it, so there you have it.</div>
<div style="text-align: justify;">
For the sake of completeness: if you run this code inside a container your results may vary. Depending on libraries used and/or your container, other implementations could have been registered which are then used rather than Oracle's internal one (see URL.setURLStreamHandlerFactory()). So now you might be thinking; why is that guy using the default HttpURLConnection then? Does he also drive to work in a wooden soapbox and cut's his grass with scissors? He could better start a fire and use smoke signals instead! Well, I can't blame you for thinking that, but we never deliberate decided on doings this. Our web service in question is a bit special and uses SAAJ SOAPConnectionFactory which on it's turn uses HttpURLConnection, which reverts to the default impl if no one registered another one.<br />
If you use a more managed WS implementation (like Spring WS, CXF or JAX-WS impls) they will probably use something completely like Apache HTTP client. And of course if you, yourself would make HTTP connections you would opt for the same. Yes, I'm promoting Apache commons HTTP client, that little critter which changes public API more often then an average fashionista changes shoes. But don't worry, I'll stop ranting now.</div>
Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-2994128444507623602014-07-24T12:03:00.001+02:002014-07-24T21:19:29.873+02:00Home Automation (Part I)<div style="text-align: justify;">
Being a developer with background
in electronics, I sometimes miss the physical aspect of writing
software. I don't mean physical like touching other people or something,
no, more like writing code that does something physically. Like
operating a lamp or a motor. So, some time ago I saw my chances fit for a
new side project: home automation.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While
investigating what I would need, I Quickly became convinced that
besides programming an installing it myself, the biggest challenge would
be to get all bits and pieces together. There are a lot of systems and
vendors and it is hard to get some objective information, or just
information all together. So I decided to start sharing my experience
here. Maybe it might serve to some good for other brave souls crazy
enough to try the same thing for their homes.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But
first, lets start with a good old rant. Why o why is the world of home
automation so badly organized? Let me come again; why? If you enter
this world, set aside all open source principles. Do not hope that you
are going to find centralized/useful documentation on how to do things.</div>
<br />
<div class="separator" style="clear: both; text-align: justify;">
Imagine
going back in a time machine, where the Internet is not sometimes
referred to as the 'interwebs'. Where the Internet is still delivered
over slow landlines owned by evil telco's wanting nothing more then your
small fortune. Where marketeers did not yet understand that a platform
can be offered almost free of charge and that there are plenty of ways
to make money. Imagine a world where not everyone speaks German. </div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div style="text-align: justify;">
German you say? Yes, get used to words as 'Schnittstelle' (not to be confused with the well known <a href="http://en.wikipedia.org/wiki/Schnitzel" target="_blank">Schnitzel</a>) or 'Zählerfernauslesesystem'.<a href="http://4.bp.blogspot.com/-HUCRb_zfK-g/U9C2nm8DvNI/AAAAAAAAAuk/pxUZmB1dl1s/s1600/Schnitzel-201020474666.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://4.bp.blogspot.com/-HUCRb_zfK-g/U9C2nm8DvNI/AAAAAAAAAuk/pxUZmB1dl1s/s1600/Schnitzel-201020474666.jpg" height="140" width="200" /></a>
Apparently a lot of these companies are German, and while most
documents are indeed also available in English, some are left
untranslated so you are stuck with the German version. Fact is that most
of these sites need to be browsed in English and a second time in
German locale to make sure all information surfaced. Some documents
simple don't appear otherwise. But it is what it is, so lets suck it up
and proceed.</div>
<br />
<div style="text-align: justify;">
As
with every project it all starts with the requirements. This time being
the customer myself, having bad, or no, requirements is not going to be
an excuse. But here also lies the danger. Home automation is very
subjective. Some people want their eggs to be backed automatically in
the morning, others just want a panic button. For me it is a subtle
mixture between what is doable, what is actually needed and the budget.
The must haves:</div>
<ul style="text-align: justify;">
<li style="text-align: left;">All light points in the house should be
able to be programmed and controlled via software. And with all lights I
mean all of them, no exceptions</li>
<li style="text-align: left;">Some power circuits needs to be controllable</li>
<li style="text-align: left;">Lights in hallway, toilet, bathroom,
garage, storage space, outside side door, outside house need to be
operated with motion and or presence sensors. These sensors also needs
to be cat proof. I don't want to turn my house into a blinking Christmas
tree if my cats decide to play at night</li>
<li style="text-align: left;">Buttons need RGB LEDs. If the toilet is
occupied I want to see a red light on the switch. Depending on the mode a
'sensor controlled light' is in, I want this being reflect by the LED.
The brightness of the LEDs needs to be adjusted as well depending on the
available light</li>
<li style="text-align: left;">Light sensors with timer functionality.
For example; the roof lights need to turn on when it's getting dark, but
only after 1900h. If it gets dark during daytime (bad weather or
something) the lights should not switch on</li>
<li style="text-align: left;">Scene support: lights in the living-room/kitchen need to be dimmed to preset scenes</li>
<li style="text-align: left;">Blinds for the windows need to be
controlled automatically.They need to be controlled together (the
default) but also individually</li>
<li style="text-align: left;">Ventilation. If the unit would be to loud at some point, it must be able to be controlled remotely, preferably by a phone</li>
<li style="text-align: left;">Support for programming over Ethernet</li>
<li style="text-align: left;">API for controlling the entire system via software. For example an app, or a dedicated touch panel </li>
<li style="text-align: left;">The interconnection of the components should be using a bus architecture with free topology</li>
</ul>
<div style="text-align: justify;">
Nice to haves:</div>
<ul style="text-align: justify;">
<li>Heating. This is a bit of a gray zone, but it at least the possibilities needs to be investigated</li>
<li>Metrics for gas/electricity/water</li>
</ul>
<div style="text-align: justify;">
For me this is where the
barrier between useful and affordable automation ends. Audio for
example, sure nice things are possible, but this is better controlled
separately. There is plenty of choice in media system with Spotify
integration at far better prices. The thing is that for audio one wants
to have freedom of choice and not be bound to a single (expensive)
product.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Next the hardware. Easier said then done, so first let's try to identify some categories</div><p>
<b>PLCs</b>
<br /><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-LLRfyEmhxVs/U9ANMD0UyiI/AAAAAAAAAtQ/D0I2KgFxWYo/s1600/S71500.JPG" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-LLRfyEmhxVs/U9ANMD0UyiI/AAAAAAAAAtQ/D0I2KgFxWYo/s1600/S71500.JPG" height="300" width="400" /></a></div>
<div style="text-align: justify;">
A barebone PLC, for example the Simatic S7 from Siemens. They are the
SAP of automation. You can build nearly anything with them and they are
very robust/stable. <br />
However, the thing is that they are barebone. For example, programming
is done with STL, which is very low level, comparable to assembler. On the image below you can get an idea how this looks.</div>
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-SSWP0DXzKRs/U9ANd_9ICXI/AAAAAAAAAtc/CKlqiiBj6No/s1600/lst.gif" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://3.bp.blogspot.com/-SSWP0DXzKRs/U9ANd_9ICXI/AAAAAAAAAtc/CKlqiiBj6No/s1600/lst.gif" height="263" width="320" /></a></div>
<div style="text-align: justify;">
Agreed,
you can also opt for a graphical way of programming, like LAD (ladder)
but it remains pretty low level. But that's not all. These things have a
high cost, which is understandable if you see in which kind of
factories they are being used.</div>
<div style="text-align: justify;">
There are also other manufacturers which are S7 compliant, such as VIPA, but the price still remains high.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But
the biggest downside is probably that there are no ready build HOME
components. If you want a push button with LEDs, no problem, but then
you have to build it yourself. I like a challenge but this would be a
bit too much for me.</div><p>
<b>Mini PLCs</b><br /><br />
Siemens Logo! is such an example<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-h5_Km30U1TQ/U9AOZKG5mYI/AAAAAAAAAtk/8xl-d4k2e8Q/s1600/logo%21.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-h5_Km30U1TQ/U9AOZKG5mYI/AAAAAAAAAtk/8xl-d4k2e8Q/s1600/logo!.jpg" height="320" width="320" /></a></div>
<div style="text-align: justify;">
While
they are cheaper, they still suffer the same drawbacks as its bigger
brothers. There are also no real home components and having something
simple as LED feedback on a button (e.g. a 'occupied' LED) means you'll
have to run an extra wire and an need an additional output just to steer
this one LED.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
While
there is profibus (really expensive, but ok) for Simatic, Logo! has no
bus system. This means that if you want 16 switches you'll at least need
17 wires running in your 'data' cable. Double that amount if you want
to use LEDs</div>
<br />
Don't get me wrong; Logo! (or PLCs)
do have a place in home automation. First there is the ease of
installation. You don't need cross switches or wiring switches directly
to switching points as with a traditional installation. It will be well
suited for basic stuff like push buttons (instead traditional toggle
switches), programming panic buttons, simple motion detectors etc They
are also more favorable over contactors/relays as they dynamically
programmed and prices will be almost the same.<br />
<br />
<b>Home automation PLCs</b><br />
<br />
Then there are what I call 'home automation PLCs, one of them is the Loxone Miniserver<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-RMzvSayAJWM/U9AQhGOnHzI/AAAAAAAAAuQ/fhsfHoh7FDo/s1600/loxonems.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://4.bp.blogspot.com/-RMzvSayAJWM/U9AQhGOnHzI/AAAAAAAAAuQ/fhsfHoh7FDo/s1600/loxonems.png" height="266" width="400" /></a></div>
<div style="text-align: justify;">
It
is a CPU just like Logo! or Simatic, but more focused on home
automation out of the box. It comes shipped with a webserver and phone
app allowing instant remote access from your browser/phone. There are
also several already build components which you can use straight away.
And most of all, it comes with KNX/EIB support (more on that later).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But
to use it as the main home automation solution it would suffer the same
disadvantages as the previous ones; limited intelligent components(°)
and no bus system.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
(°) limited, but there are at least extensions like a an IP cam based intercom, RGBW controllers, wireless extension and more (<a href="http://www.loxone.com/enuk/products/extensions/extension.html%29" target="_blank">check it out</a>) these home components are not available with traditional PLCs</div>
<br />
<b>Real home automation</b><br />
<br />
<div style="text-align: justify;">
With
"real" I don't mean that these system would be more stable or mature,
not at all. But they are more home oriented and therefore a closer match
as to what one would need for home automation.</div>
<br />
<a href="http://4.bp.blogspot.com/-ibJKw_ChatA/U9C3yq_t_wI/AAAAAAAAAuw/1n1lu80aK1o/s1600/nhc_3.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ibJKw_ChatA/U9C3yq_t_wI/AAAAAAAAAuw/1n1lu80aK1o/s1600/nhc_3.jpg" height="200" width="136" /></a>To make it easy I divided them in the closed systems; such as <a href="http://www.niko.eu/enus/niko/products/niko-home-control/" target="_blank">Niko</a> (Belgian company) home control (NHC) <a href="http://www.bticino.com/" target="_blank">or BTicino</a> (Italian) etc. These are examples of a closed standard and only the manufacturer in question produces components for it.<br />
<br />
<br />
<div style="text-align: justify;">
<a href="http://2.bp.blogspot.com/-5osMgzlZVCU/U9C6pxfkASI/AAAAAAAAAu8/Qlqfe_knaCQ/s1600/basalte-deseo-new-1.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://2.bp.blogspot.com/-5osMgzlZVCU/U9C6pxfkASI/AAAAAAAAAu8/Qlqfe_knaCQ/s1600/basalte-deseo-new-1.jpg" height="127" width="200" /></a>Then there are the open systems such as <a href="http://en.wikipedia.org/wiki/KNX_%28standard%29" target="_blank">KNX</a>/<a href="http://en.wikipedia.org/wiki/European_Installation_Bus" target="_blank">EIB</a>.
The latter is an open standard and every one is free to build
components compatible with this standard. This means that you don't have
a vendor lock-in; you can mix different brands and components, as long
as they are KNX/EIB compatible. For example, one can have a <a href="http://www.mdt.de/EN_Push_Buttons.html" target="_blank">MDT pushbutton</a> combined with a <a href="http://www.jung.de/en/online-catalogue/69799111/" target="_blank">Jung dim actuator</a> and a <a href="http://www.basalte.be/en/switches" target="_blank">Basalte thermostat</a>. Just as NHC, KNX uses a bus (EIB) with a free topology.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Ok,
then what is the best solution? There is no objective answer in which
is better. It basically depends on budget, taste and mindset. For
example; vendor lock-in is mostly not a good idea. But if you chose a
good vendor that "has it all" it can also have advantages. If you go to
your local Niko dealer, someone can probably explain all components. You
get everything (information, ordering, support) from a single place. No
need to endlessly browse the Internet comparing manufacturers or
products. On the other hand a closed standard might impose future risks:
what if Niko decides to stop producing NHC? They already did that with
it's predecessor: Niko bus. Agreed, they solved it with NHC/NB bridge
which allows to add NHC components to an existing NB installation. But
not all functionality is usable this way. So if you bought Niko bus
some years ago you are already limited in adding new components.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
KNX
excels in the fact that it has no central processor (unlike PLCs). It
is message driven and each sensor/actor has it's own processor. They
communicate with each other using telegrams (~messages). A sensor or an
actor generates can subscribe to a so called 'group address' (like a
multicast address) and it will then be able to pick up telegrams sent to
that address by other sensors or actors. Based on the configuration
they can decide to do something with the telegram or ignore it. But
there lies also it's weak point: each
component is responsible for generating or processing telegrams, but it is
limited to the logic offered by that component. For example; a light
point needs to be switch on conditionally based on the value of a light
sensor and the wall-clock time. This clearly involves clock logic. While
perfectly possible with KNX, you'll need a separate actor which is able
to generate telegrams based on clock schedules. This starts to become a
pricing issue as each KNX actor is a processor on its own making them
not cheap.</div>
<br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-BViXHysPVlk/U9C8yNuYQkI/AAAAAAAAAvI/Es6k5y5569U/s1600/gira.jpeg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-BViXHysPVlk/U9C8yNuYQkI/AAAAAAAAAvI/Es6k5y5569U/s1600/gira.jpeg" /></a></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
But there is also a solution. The so called KNX/EIB 'servers'. They allow more advanced and centralized programming and form an extension to the traditional KNX sensors and actuators. An example of this is Gira's <a href="http://www4.gira.com/en/gebaeudetechnik/produkte/neuheiten/homeserver-update.html" target="_blank">home server</a>. However, it's price tag is really, really high
at €2.200. Luckily,
this is where Loxone comes in. Loxone offers the same functionality but
at a reasonable price. It can even do more as it is a traditional CPU
with KNX/EIB integration. To program your KNX components via Ethernet,
you also need an IP gateway. This is again an extra (special) actor.
With Loxone you get all of that combined in a single module. So by
adding Loxone we get follow things in return:</div>
<ul>
<li>A 'soft entry' into home automation system. It allows for remote
access via an API and apps thanks to it's built in webserver. Since it's
KNX/EIB compliant, Loxone can be used as the programmatic interface to
EIB, not needing extra KNX components</li>
<li>KNX/EIB gateway; using Loxone one can program KNX/EIB components using it as a gateway</li>
<li>The inputs on the Loxone are not really usable when using EIB.
However, the outputs are. By using them you are also saving a 8 digital
output KNX actor</li>
<li>Actual programming with logic circuits, controllers, smart components, you can even program in pico C if that would be required</li>
</ul>
<br />
<div style="text-align: justify;">
A KNX gateway and
8DO KNX actor already cost more then the entire Loxone setup. So it's a
win-win deal: less money for more functionality.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So
in the end I decided that an open system (such as KNX/EIB) in
combination with a home automation PLC (Loxone Miniserver) combines the
best of two worlds. KNX/EIB offers an open standard and freedom of
choice in vendors and components. It also decentralized, message driven
and open ended architecture which I felt comfortable with. Loxone adds
the central processing power and interface to other platforms
(web/mobile). It also allows to program all logic you would even need
for your home automation at a reasonable price.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
There
is however one thing to keep in mind. While Loxone Miniserver is
KNX/EIB compliant, it cannot be programmed like other KNX components. In
other words, Loxone is programmed separately with different software
and different 'programming techniques'. But this actually makes sense.
The way a KNX/EIB is programmed does not lend itself for complex logic.
After all, KNX/EIB programming is more 'configuring'.</div>
<div style="text-align: justify;">
The
Loxone development environment is better suited for this, taking full
advantage of its possibilities. Do note that a Gira Home server for
example is also programmed in a different way than traditional KNX/EIB components, so this is just
natural. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In
the next part I'll show the components I'm planning to use to meet the
requirements and the test board together with the initial programming I
did to be able to test everything from my desk.</div>
<br />Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com5tag:blogger.com,1999:blog-3406746278542965235.post-48073960992235923432014-01-28T16:38:00.000+01:002014-01-28T17:11:56.216+01:00Some Of My Mostly Used Development Mantras (1/2)<div style="text-align: justify;">
<b>Statement: I assume it works this way!</b></div>
<div style="text-align: justify;">
<b>Mantra: assumption is the mother of all fuckups</b></div>
<div style="text-align: justify;">
<br />
<a href="http://4.bp.blogspot.com/-fJR2k3J3LwI/UudUYlw73DI/AAAAAAAAAl0/vCkHrKs8vaQ/s1600/us2.jpeg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://4.bp.blogspot.com/-fJR2k3J3LwI/UudUYlw73DI/AAAAAAAAAl0/vCkHrKs8vaQ/s1600/us2.jpeg" height="228" width="320" /></a>Let's start with this short fragment from Under Siege2: <a href="http://www.youtube.com/watch?v=wg4trPZFUwc" target="_blank">http://www.youtube.com/watch?v=wg4trPZFUwc</a></div>
<div style="text-align: justify;">
<br />
In this case his assumption had a rather lethal outcome which you might probably remember if you have seen the movie. But ok, being a mercenary, his situation might seem different from a day to day developer. So going further with this example, you might remember there was also a computer whiz on the train doing stuff like operating the satellite. Now imagine you're that guy and your asked to startup satellite communications. You quickly write some code and in doing so you write this line: </div>
<pre class="brush: java;">i = i++;
</pre>
<div style="text-align: justify;">
Normally we would write something as 'i++' leaving out the assignment. However, the assignment seems redundant at most. We assume that it does not has any other side effect and we startup our program. But then suddenly....BOOM! Communication fail, Casey Rayback gets his ass kicked (actually I would love to have seen this), nukes are launched and world peace is far away. Our assumption had a rather negative impact.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Ok, let's see what this is about and make a small test. What do you expect it prints out, do you have any idea why it does so?</div>
<pre class="brush: java;">int i = 0;
i = i++;
System.out.println("i:"+ i);</pre>
<div style="text-align: justify;">
While you're thinking about this, let me explain what happens. As can be seen we have an expression, existing of a unary operator and an assignment. As discussed before the assignment is redundant. But besides being redundant it is also causing an unpleasant side effect. It makes this code behave differently then one would expect. If we drop the assignment (just leaving 'i++;') the code would exactly do what you think it would do.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What happens is that when using an assignment, the JVM will put the value of the operands on the so called operand stack before doing something with them. When the expression is evaluated, the value for making the assignment to the left hand variable is popped from the operand stack. That sneaky unary operator follows different rules. It uses the 'iinc' instruction. The iinc instruction takes as parameter a position in the local variable array of the current frame to put it's value.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So, what happens is that "0" is pushed on the local operand stack. Next the variable "i" is incremented by 1 because of the iinc instruction. At this time the local variable 'i' would actually have the value '1'. We can't show this in a normal way, since the assignment/expression is evaluated in a single step. When the assignment is made, it will pop the latest value from the operand stack assigning it to the variable 'i'. This means that 'i' will be overwritten with the value from the operand stack, changing it's value from 1 back to 0.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<i>Note: when you write 'i = <b>++</b>i;' thinks will work as expected and the assignment is just redundant without side effects. In that case the iinc instruction is executed before pushing the local variable value to the operand stack. The rational is actually no different as when incrementing a variable in a method call; someMethod(++i) or someMethod(i++) . However, with an assignment it seems less clear that a pre or post unary makes any difference, because we tend to relate it to i++ or ++i on a single line where it absolutely makes no difference if the ++ is pre or post.</i></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In nearly every line of code (depending on experience and knowledge) we take assumptions. This works out most of the time because some of the assumptions will be correct. Incorrect assumptions do not necessarily lead to bugs as they might get balanced out by other assumptions. Or, maybe the side effects are never detected because of the boundaries in which our software operates.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
A good way to track down incorrect assumptions in code is to write tests. But that alone is not enough. It is also important to accept this rule. No matter how brilliant you are, there is always (a lot) of internals you don't know or fully understand. One needs to remain skeptical, don't be lazy. Always try lookup things as much as possible. Even in the case when there is only a little bit of doubt: look it up and/or make small test case. It does cost time and energy, but the ROI (being able to write working and high quality code) will be well worth it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Statement: don't worry, it's only temporary!</b></div>
<div style="text-align: justify;">
<b>Mantra: there is no such thing as temporary code</b><br />
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Ji9eu9KoR0w/UufOFIlUNZI/AAAAAAAAAmE/dObLgJfWVuk/s1600/westvleteren.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-Ji9eu9KoR0w/UufOFIlUNZI/AAAAAAAAAmE/dObLgJfWVuk/s1600/westvleteren.jpg" height="320" width="240" /></a></div>
</div>
<div style="text-align: justify;">
Agreed, a lot of things in life are temporary. The<br />
stock of Westvleteren 12 in my fridge is just one example. But in software projects, "temporary" tends to start a life on its own inflicting poorly designed or written code along its way.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This makes sense if you think about it; I don't think anyone works with the same amount of energy or dedication if something is considered temporary. The problem is however that temporary code will become permanent before you even realize it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
An example I can share on this topic is a user management system which we designed for a previous project. The system was going to be a "temporary" user management system because the "permanent" system was delayed for some reason. I won't go into detail, but the name of the software needed to reflect that it was temporary. So it was named TUM (Temporary User Management system). And you guessed it; in the end the system existed for years to come and over time became the one and only UAM. After a while it even became hard to convince new colleagues that the "T" actually stood for "temporary".</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I'm going to put this a bit harsh, but "temporary" helps no one and gives developers a wild card to justify bad decisions, laziness or incompetence. In such situations chances are high that the design will not be what could have been and the resulting code will be suffering quality issues. So the rule is simple: always design or write code in the assumption it will stick around for quite a while. It will be beneficial in the long run as nearly all code will, in one form or another.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
But make no mistake friends, no matter how dedicated or how good you are, the dark side is always waiting around the corner. If you are being tempted, simply ask the question how you would feel if your car mechanic decided to fix your brakes "temporary" because he didn't have any idea at the time on how to fix them properly? Enough said I think.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Like most rules, there are exceptions. It would be naive to think that there are no real temporary software projects in general. Of course there are, and fundamentally it is no big issue if that would be the case. The important part is to keep this decision on project management level. Doing so one avoids infecting the team with it.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Don't forget that in the end you're never sure how temporary a project is going to be. And even if the project is really temporary, it's code might still find it's way to other modules or projects in the future.</div>
Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-60648543649566902482014-01-04T17:15:00.000+01:002014-01-05T21:06:09.380+01:00Git: Keeping Your History CleanAs we migrated lately from subversion to git, it was clear from the beginning that we needed some extra rules for keeping the history clean. The first thing that we stumbled upon were the so called automatic 'merge commits' performed by git.<br />
<br />
In case you don't know
what they are; imagine a typical closed project setup with a central repository to share work between multiple developers. Now suppose developer John committed and pushed some changes. In the mean time there is Jim developing something on the same branch on his local repository. Jim has not pulled from central, so the central repository is still one commit ahead. If Jim is ready with his code he commits the changes to his repository. When he now tries to push his work to central, git will now be instructing Jim to pull first (to get the
latest changes from central) before he'll be able to push again. From the moment Jim
does a "git pull", a merge commit will be made by git indicating the commit done by John.<br />
<br />
This is new when coming from
subversion as with svn you won't be able to commit in the first place.
You would first have to "svn up" before doing "svn commit", so your
changes would still be in your local working copy (unlike with git where
they are already committed to your local repository). To be clear: these merge commits don't
form a technical issue as they are "natural" (to a certain extend) but
they do clutter the history a lot.<br />
<br />
To show how this would look in our history, we'll simulate this locally using three repositories (central, a bare git repo and john/jim both 2 clones of central). First John commits and pushes john.file1. Then Jim creates and commits jim.file1. If Jim now tries to push his work, he'll be getting:<br />
<pre class="brush: bash;">
! [rejected] master -> master (fetch first)
error: failed to push some refs to '/tmp/central/'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first merge the remote changes (e.g.,
hint: 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details
</pre>
If Jim would simply 'pull' the changes from origin would be merged and Jim will on his turn be able to push his changes to central. All good. However, our history will now contain the so called merge commit as highlighted below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-AKWo1f-wTNY/Usmr69mijrI/AAAAAAAAAlg/XR8e85upgqw/s1600/git1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="344" src="http://3.bp.blogspot.com/-AKWo1f-wTNY/Usmr69mijrI/AAAAAAAAAlg/XR8e85upgqw/s640/git1.png" width="640" /></a></div>
<br />
As said before, this
is happening because the local branch has advanced, compared to central, with the commit of Jim. Therefore git can no longer do a fast forward. Git will (by default) basically be
merging in the changes it received from origin and thus has
to make a commit. Altering the commit message is still possible, but
you'll end up with a actual commit either way. The result is that the
change appears to be done twice when looking at the history which can be
quite annoying. Also, if the team is committing/pushing fast (as they should) a lot of
these entries will appear and really start to clutter the history.<br />
<br />
A solution for this is to rebase instead of merge. This can be done by doing:<br />
<pre class="brush: bash;">
git pull --rebase
</pre>
When using 'git pull' git will first bring the changes from origin to the local repo and then (by default) merge them into your local working copy. If rebasing instead, git would (temporary) undo your commits, fast forward your branch and the re-apply your changes. These commits will be new commits as the old ones are thrown away. They get a new SHA1 identifier and all the likes.<br />
<br />
Rebase is however only a solution in particular scenario's, so we would certainly not make it the defaul for everything. From the <a href="http://git-scm.com/book/en/Git-Branching-Rebasing" target="_blank">Git book</a> :<br />
<br />
<i>If you treat rebasing as a way to clean up and work with commits before
you push them, and if you only rebase commits that have never been
available publicly, then you’ll be fine. If you rebase commits that have
already been pushed publicly, and people may have based work on those
commits, then you may be in for some frustrating trouble. </i><br />
<br />
As long as you are not playing with commits that have already been pushed out to others there is no problem. We are working in a standard closed Java project with a single central repository that is adhering to these rules:<br />
<ul>
<li>Developers don't synchronize repositories directly with each other, they always go via central</li>
<li>Developers don't push to anything else then central</li>
</ul>
In this case there is no risk for commits to be made public before rebasing them using git pull. Doing so we can safely make 'git pull' rebase automatically for our project master branch using:<br />
<pre class="brush: bash;">
git config branch.master.rebase true
</pre>
Another problem that caught my attention (being triggered when looking into the automatic merge commit problem) is that our
branching structure was not optimal. We are using a feature branch approach but as
feature branching (and merging) with subversion comes at high pain we only
did it if really necessary. However, when using git creating branches
and merging is just a breeze so we changed our rules to be more natural:<br />
<ul>
<li>There must be a (in our case; more) stable master branch on which
only completed features will be merged into. Theoretically speaking, no
one works directly on the master. This implies that almost all work is
executed on feature branches</li>
<li>Releases are made from a release branch which is a copy of the master branch at the point the release is triggered</li>
<li>The master log must be as clean and straightforward as possible:
each feature being merged back into the master must be exactly one
commit with the log message stating the ticket and the history showing
all changes relevant to that feature.</li>
</ul>
When working like this chances that you get automatic merge commits are very minimal. Reintegrating features branches would become an (almost) atomic
operation avoiding (nearly) any risk for having those automatic merge
commits, even when not rebasing. Of course, some rules must be followed before integrating:<br />
<ul>
<li>Code reviews must have been done on the feature, one way or the other </li>
<li>The feature branch must first be updated with the latest version of
the master. This happened regularly while developing the feature (you
want to be as up to date with the master as possible) but certainly a
last time before you're planning to reintegrate the feature. With git
this means you simply do "git merge master" while on the feature branch
and it will complete in matter of seconds. Doing so conflicts (of other
features that have been reintegrated in the master in the mean time) can
be resolved while on the feature branch.</li>
<li>Tests and the likes must be run on the feature and must be 100% success before reintegrating the feature.</li>
</ul>
When following these rules the following 3 lines will reintegrate the feature succeeding one after the other making it an almost atomic operation and thus practically making rebase unnecessary:<br />
<pre class="brush: java;">git checkout master
git pull
git merge --no-commit --squash >feature branch<
git commit -am "FEATURE-XYZ"
git push
</pre>
Now, even when working like this we leave rebase set to true for the master as it is still possible for the remote to change in the meantime. In the event this happened (after committing the merge) a simple 'git pull' (with implicit rebase) will solve the problem so that our push will succeed.<br />
<br />
Other important rules for keeping clean history are making use of '--squash' as used above:<br />
<br />
The --squash will drop any relation to the feature it came from. Doing you can create a single commit that contains all changes without taking any other history from your feature into your master. This is mostly what you want. A feature can contain a lot of "junk" commits which you don't want to see on your master. In the typical case you simply want to see the result and not the intermediates. Also note that --squash also implies --no-ff as git will not fast forward even if it's possible (otherwise you would not be able to get a single merge commit in the first place).<br />
<br />
To illustrate this, when we would simply be reintegrating using 'git merge' the history would look like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-bLlvjWXN5Ic/UslRhZPgttI/AAAAAAAAAk4/DymgQfEifCc/s1600/git2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="344" src="http://3.bp.blogspot.com/-bLlvjWXN5Ic/UslRhZPgttI/AAAAAAAAAk4/DymgQfEifCc/s640/git2.png" width="640" /></a></div>
<br />
Since the feature was updated with the master before reintegrating, git performed a fast forward. If you look at the history the commits from the feature will be in-lined into the master. This gives the impression that all work had been done on the master. We can also see all intermediate commits and other non interesting commits such as the merge commits from updating the feature with the master. Not a real problem, but to see what has been changed you would (possibly) have to go through a lot of clutter and look at each individual commit to get the big picture, which is not what you typically want.<br />
<br />
When disabling fast forward (git merge --no-ff) the history would look like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-XERxy4mLwM4/UslSaTsbZ7I/AAAAAAAAAlA/7fEOjtznmG8/s1600/git3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="344" src="http://4.bp.blogspot.com/-XERxy4mLwM4/UslSaTsbZ7I/AAAAAAAAAlA/7fEOjtznmG8/s640/git3.png" width="640" /></a></div>
<br />
A bit better as we can now see which commit are related to feature-xyz. However, the changes are still scattered over different commits and we still have other commits in which we're not interested in. Someone needing history of feature-xyz is normally not interested in how the feature evolved or how many times a class was refactored. Typically you're only interested in the end result: the changes/additions that will be added to the master branch. By adding "--squash" we accomplish a clean and single commit in our master which list all changes for that feature:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-S1oM0hrY8M0/UslS1NN_IcI/AAAAAAAAAlI/xJT1oT8eKsg/s1600/git4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="344" src="http://3.bp.blogspot.com/-S1oM0hrY8M0/UslS1NN_IcI/AAAAAAAAAlI/xJT1oT8eKsg/s640/git4.png" width="640" /></a></div>
<br />
Note that the history of the feature is not lost of course. You can still go back the feature branch and look at the history if that would be necesary.<br />
<br />
The --no-commit is convenient if you are skeptical (like me) and want to review to changed/added files for a last time before making the reintegration final. If you leave this option out Git will directly commit the merge without leaving you time to do this final check.<br />
<br />
On a side note: to perform code reviews on features before they can be reintegrated I tend to create a (local only) branch of the master being a kind of 'dry run' branch. The feature is then reintegrated on that review branch (without committing changes) just to perform code review. Doing so all changed files will be marked dirty by your IDE making it easy to see what has actually changed (and to compare them with the HEAD revision). Then I gradually commit changes as I reviewed them. When code reviewing is done, the merge will be completely committed on the the review branch and it can then be merged back to the feature branch before it's finally reintegrated into the master branch. I (normally) don't make the review branch remotely visible, and just delete them locally after reviewing is done. Theoretically one can also merge the review branch back to master directly, but since this review branch will be deleted the real feature branch would never got the changes performed by the code review and you would be losing that history.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-92129787150341691992013-11-15T12:28:00.001+01:002013-11-16T10:22:58.874+01:00Oracle Please Stop Shipping Ask.com ToolbarLately I configured a JRE on some windows machine.
It was not a developer machine and the goal was to run some Java, so I installed the JRE. Normally I download everything from oracle.com, but I was lazily this time and google presented java.com as the first result. I never used java.com. I know Java is from Oracle, but the paranoia routine in my brain has not yet white-listed this site as having something to do with Oracle and as such I refuse to associate it with them. But hey as I said, I was lazy and the site seemed legit, so I started downloading.<br />
<br />
While going through the install wizard, I suddenly got this screen:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-JGeX3MC4HSg/UoYEcTNzi-I/AAAAAAAAAaY/sEABtL2loLg/s1600/asktoolbar.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="http://4.bp.blogspot.com/-JGeX3MC4HSg/UoYEcTNzi-I/AAAAAAAAAaY/sEABtL2loLg/s400/asktoolbar.png" width="400" /></a></div>
<br />
<br />
I was baffled to see this. The first thing that went through my mind was: f*** ! I must have downloaded a compromised installer! hAx0orz in my interwebs! So I immediately executed evasive maneuvers theta 3 for windows users: No time for checksums! Pulled out the power, ripped out ram and hdd and fried them instantly.<br />
<br />
After I was calmed down I was curious in finding out which compromised installer I had downloaded. So I used a working machine to go out on the Internet.
It turned out I was in for an even bigger surprise. It's just part of the official installer! <a href="http://www.java.com/en/download/faq/ask_toolbar.xml" target="_blank">http://www.java.com/en/download/faq/ask_toolbar.xml</a><br />
<br />
Even more, this is not even breaking news as it has been around there for quite some time (and I'm not the first one writing about this). It always escaped me since it's only in the JRE, and only in the one you download from java.com *sigh*. The JDK and the JRE exe's comming from oracle.com don't contain this crap.<br />
<br />
This is bad. Really bad. Yeah, ok, you can disable it on install, true, but that won't cut it.
The fact that it is only in JRE installs via java.com makes it even worse. People working professionally with Java (requiring a JDK install instead) could see through this if it was in that type of installer (but then it would probably don't generate profit either).
However, systems using JRE's are end user systems (or end user systems managed by other professionals).
These are the people working with the applications we write. If these people starts to associate Java with crapware then we have a problem.<br />
<br />
It might well be that this was a decision taken by SUN and that Oracle now has to live with the contract, but then it's time they end it. This should stop as it's damaging the reputation of Java all together!<br />
<br />
I kindly invite you in joining me signing this petition, at least if you also agree this should stop: <a href="https://www.change.org/petitions/oracle-corporation-stop-bundling-ask-toolbar-with-the-java-installer" target="_blank">https://www.change.org/petitions/oracle-corporation-stop-bundling-ask-toolbar-with-the-java-installer</a><br />
<br />Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-81051956421075737272013-10-29T16:55:00.001+01:002013-10-29T19:06:47.725+01:00Building a SOAP Webservices Proxy Module using Spring WebservicesSome time ago I wanted to see how easy it is to write a web services proxy (wsproxy) using Spring Web Services.
So, I thought I would share the result on Github. <a href="https://github.com/koen-serneels/ws-proxy/" target="_blank">Feel free to use it</a> (Apache v2 license) or let it serve as basis for your own development.
The remainder of the article will explain the idea, how I used Spring web services to build it and a short guide on how to use the current implementation.<br><p>
A wsproxy in this context is a central soap aware access layer which relays messages between systems.
This relaying works in two directions; for services hosted internally (inbound mode) but also for services hosted externally were we are the client (outbound mode).
One can compare this with a traditional http forward/reverse proxy but instead of operating on application transport it goes one level higher in the stack and deals with application messages; in this case soap.<br><p>
In outbound mode our internal services will (normally) use the wsproxy as a http forward proxy. It will then deal with delivering the received message to the actual target. For inbound mode the module will act as a reverse proxy accepting incoming messages from external clients and relay them to our internal services.<br><p>
Before we continue, a few words about forward/reverse proxies:<br><p>
A forward proxy is something the client explicitly configures in addition to specifying a target URL. The http stack of the client will send the message to the configured proxy and sent the host/port of the actual desired target host via http headers (Host header). The proxy then deals with forwarding the request to the desired target host/port and return the response to the client. Thus the proxy composes the host and port for the target URL out of the http Host header. For the path it will use the path from the request as received from the client. A reverse proxy behaves (from the point of the client) as the actual target server. The client is not aware of any proxies and does neither have to configure anything special. What the client uses as target URL is the URL of the reverse proxy. The reverse proxy will be able to intercept the message from the client and forward it to the actual target within the network. The reverse proxy will required extra configuration as for example the URL (host, port and possibly path) of the target service, as host/port cannot be deduced from the request or http headers.
<br><p>
When using this module, one can still implement an actual http reverse proxy on boundaries for TLS offloading or other transport purposes. For external inbound traffic (messages coming from external clients destined for internal service) the wsproxy module will simply be the second reverse proxy in line for incoming requests. For internal outbound traffic (messages from internal clients destined for external endpoints) the URL of the http reverse proxy will be configured as the target URL for that specific service on the wsproxy module.<br><p>
One of the features of having such a soap aware wsproxy is centralizing concerns. The goal is that soap traffic passing through can be intercepted. This way we can implement things such as: audit logging, monitoring, access control, message security, ... doing so we create a centralized place to deal with these requirements instead of having to re-implement them per application.<br><p>
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-_M5eEYCZvqM/Um0nlc1ahSI/AAAAAAAAAY0/ZOXfRzDIJsc/s1600/wsproxy-usages.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-_M5eEYCZvqM/Um0nlc1ahSI/AAAAAAAAAY0/ZOXfRzDIJsc/s640/wsproxy-usages.png" /></a></div>
<p>
I chose Spring web services because there is no complicated infrastructure or design to understand. It it also very extensible and offers re-use at the right level for our requirements.<br> However, at this time I should also point out that there are existing solutions out there which can do this as well and more. They often go under the name of xml security gateways and come as software package or fully equipped appliances. As always you should outweigh the benefits of these existing solutions over writing something yourself. Fact is that they don't come for free (to say the least) and you still need someone with the right skills for configuring and maintaining them. As we will see, our requirements are easy to fulfill with a bit of code (and the help of Spring Web Services) giving us all the control we need.<br><p>
For the requirements I had an easy to extend design in mind with following out of the box requirements:
<ul>
<li>Outbound mode needs to be configuration free for standard usages. This means that when needing to access a new external services our proxy should relay messages without requiring extra configuration</li>
<li>Messages passing the gateway need to be logged. Without extra configuration the entire message should be logged by default. Optionally, we need to be able to configure more fine grained which parts needs to be logged for specific services</li>
<li>For (mostly external) outbound communication we should be able to configure a forward proxy or override the target with a pre-configured host (an existing reverse proxy or the actual endpoint)
<li>The module must be able to forward messages over secure transport in case there is no external reverse proxy present for offloading outbound secure transport.
For inbound secure transport we will let this be handled by the container where the module is running on; so this is out of scope for the module</li>
<li>Be able to apply and handle message integrity/confidentiality</li>
</ul>
The component design looks like this:<br><p>
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-b_r9NXwHsGM/Um0oY1zsJvI/AAAAAAAAAZA/qP2QKwP_ooQ/s1600/wsproxy-components.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-b_r9NXwHsGM/Um0oY1zsJvI/AAAAAAAAAZA/qP2QKwP_ooQ/s640/wsproxy-components.png" /></a></div>
<p>
There are 3 main components. The <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/core/CatchAllEndpoint.java" target="_blank">endpoint</a> (catch-all endpoint in the drawing), which will act as the receiver for the message being sent to the wsproxy. The <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/core/ForwardingClient.java" target="_blank">forwarder</a>, which will relay the message to the target. Finally, the interceptor chains are the hooks where we are able to intercept the messages being sent/received and do something with them.<br><p>
These 3 components are offered by Spring Web Services; the endpoint is a <i>org.springframework.ws.server.endpoint.annotation.Endpoint</i> implementing <i>org.springframework.ws.server.endpoint.MessageEndpoint</i> to be able to receive the raw payload. The forwarder uses <i>org.springframework.ws.client.core.WebServiceTemplate</i> and the interceptor chains are <i>org.springframework.ws.client.support.interceptor.ClientInterceptor</i> and/or <i>org.springframework.ws.server.EndpointInterceptor</i> depending on which sides they need to function (more on that later). For message security we will use WSS4J, but this is just an implementation of an interceptor, thus not a new component.<br><p>
It is important to realize that there are two interceptor chains. From the point of the wsproxy, we'll call the first one the "inbound chain". This is the one operating between
the client and the wsproxy. The "outbound chain" is the one operating between the wsproxy and target endpoint. So, if we have an internal client accessing an external endpoint via our wsproxy, the inbound chain will be invoked when the message is received by the wsproxy. The outbound chain will be invoked from the moment the wsproxy relays the message to the target endpoint. Spring has two interfaces to distinguish on which "side" the interceptor operates (an interceptor can also implement both interfaces, making it able to function on both sides). The <i>org.springframework.ws.server.EndpointInterceptor</i> operates on the endpoint side, for the wsproxy this is inbound. The <i>org.springframework.ws.client.support.interceptor.ClientInterceptor</i> operates on the client side, so for the wsproxy this is outbound. Btw; we're using inbound and outbound rather then the original Spring naming (client/endpoint) to avoid confusion. As you've noticed by now, the wsproxy is also an endpoint and a client. However, when we refer to "client" we mean the actual service client and the "endpoint" is the actual target service.<br><p>
The module itself will run on a standard JEE servlet container as your typical Spring application. For all inbound traffic the http (or https) connector from the container is used.
For all outbound traffic the WebServiceTemplate is used configured with commons httpclient under the hood, which we'll be able to use for both http and https if required. Service identification is done the "doclit" style. This means we take the first element of the body including it's namespace. This is represented as a <a href="http://docs.oracle.com/javase/7/docs/api/javax/xml/namespace/QName.html" target="_blank">QName</a>. This identification is important as we'll have configuration on a per service basis, such as the forward proxy, forwarding protocol, endpoint URL mapping, specific loggers etc.<br><p>
Ok enough of this. let's take this baby for a spin! Import the project in your IDE of choice, make sure you import it as a Maven project as Maven will have to filter the file <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/resources/active_environment.properties" target="_blank">active_environment.properties</a> (this is done automatically via the default profile). Then we will:
<ul>
<li>Setup a normal standalone soap based endpoint</li>
<li>Deploy wsproxy</li>
<li>Use a web services client to access the endpoint via the proxy module</li>
</ul>
For bootstrapping a simple endpoint there is a class <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/test/java/be/error/wsproxy/SimpleEndpoint.java" target="_blank">SimpleEndpoint</a> foreseen in the tests sources which uses the JDK internal JAX-WS and http server to bootstrap a webservice endpoint:
<pre class="brush: java;">
public class SimpleEndpoint {
public static void main(String args[]) {
Endpoint.publish("http://localhost:9999/simple", new SimpleWebServiceEndpoint());
}
@WebService
public static class SimpleWebServiceEndpoint {
public Date getCurrentDate(String randomParameter) {
return new Date();
}
}
}
</pre>
Just run this as a new Java application, it will remain running until the process is killed. Boot the wsproxy by deploying the project to your server of choice (I'll be using Tomcat7), there is no extra configuration required. As for the client we'll be using soap-ui (you can also use cURL if you like). In soap-ui we first have to create the project. We do this based on the WSDL exposed by our test service (accessible under http://localhost:9999/simple?WSDL). Next, we'll have to configure our wsproxy module as the http forward proxy in soap-ui:
<table><tr> <td>
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-hgujt41py4k/Um0zwczmskI/AAAAAAAAAZY/5uFWtTFaMmc/s1600/soap-ui-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; "><img border="0" src="http://2.bp.blogspot.com/-hgujt41py4k/Um0zwczmskI/AAAAAAAAAZY/5uFWtTFaMmc/s300/soap-ui-2.png" /></a></div>
</td><td><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-_TXOV0NjyuM/Um0zRkV9-6I/AAAAAAAAAZQ/Vi-7pdzYYVE/s1600/soap-ui-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-_TXOV0NjyuM/Um0zRkV9-6I/AAAAAAAAAZQ/Vi-7pdzYYVE/s300/soap-ui-1.png" /></a></div>
</td></tr></table>
The soap-ui projects are also <a href="https://github.com/koen-serneels/ws-proxy/tree/master/src/test/soapui" target="_blank">availale in the project</a> if you want. <b>Don't forget to enable the proxy settings as explained above, they are not saved as part of the project.</b> <br><p>
<b>Important:</b> don't forget to disable the proxy settings again if you are starting with a new project. soap-ui will use the proxy settings for standard http traffic and not only for soap/http. For example; when creating a new project based on a WSDL URL, soap-ui will use the http proxy settings as well to retrieve the WSDL. Since the wsproxy module is not a pure http proxy (but a soap proxy instead) it will not allow non-soap traffic through.<br><p>
The last thing we need to configure is the target URL in soap-ui. The wsproxy module is by default (on tomcat at least) deployed under the context root named after the filename. In our case this means the module is reachable under: http://localhost:8080/ws-proxy/ <br>
There are two options:
<ul>
<li>Deploy the module under the root of the application server (/) instead. In that case nothing needs to be changed to the target URL. The target URL will remain the same URL as you would be using without the proxy module</li>
<li>Use a context root of choice, but in that case you'll have to prefix the context root to the target URL</li>
</ul>
In our case we are in the second scenario, this means that we'll have to change the proposed target URL from "http://localhost:9999/simple" to "http://localhost:9999/<b>ws-proxy</b>/simple". <br><p>
What happens is that soap-ui sends the request to the host/port specified in the proxy settings (it will thus not sent the request to localhost:9999 but to localhost:8080 instead). The path however is retained; the request is actually sent to localhost:8080 with path "ws-proxy/simple". With the module deployed under "ws-proxy" you can now see why this path prefix has to be there. If the path would start with "simple" we would get a 404. The remainder of the path is not important for the infrastructure, as the Spring dispatcher servlet (configuration can be found in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/core/WsProxyWebApplicationInitializer.java" target="_blank">WsProxyWebApplicationInitializer</a>) is bound to "/*". So every subsequent path is treated by the servlet in every case.<br><p>
To be able to forward the message to the actual target, the module will calculate the target URL:
<ul>
<li>First check if there is a pre-configured target URL for the given endpoint based upon the service identification (payload root element + namespace). This is configured in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/properties/EndpointTargetUrlMapping.java" target="_blank">EndpointTargetUrlMapping</a> as we will see later.</li>
<li>If nothing found, check if there is a http Host header present, use host:port as the target server. For the path, use the path as present in the request, but subtract the context root under which this module is deployed (if any)</li>
</ul>
The latter means that in our case the module is deployed under "ws-proxy" and the request path is "ws-proxy/simple", which will result in a target URL of "http://localhost:999/simple"
When executing the request, we'll get this answer:<p>
<div style="text-align:center;">
<a href="http://4.bp.blogspot.com/-xDOoo5hgKrw/Um58bXln4uI/AAAAAAAAAZo/axgZr267wiQ/s1600/soap-ui-4.png" imageanchor="1" ><img border="0" src="http://4.bp.blogspot.com/-xDOoo5hgKrw/Um58bXln4uI/AAAAAAAAAZo/axgZr267wiQ/s400/soap-ui-4.png" /></a><br><p>
</div>
In the wsproxy log file we can see the intercepted request and response being logged:
<pre class="brush: xml;">
51948 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.interceptors.internalchain.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate INBOUND SIDE Request:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsp="http://wsproxy.error.be/">
<soapenv:Header/>
<soapenv:Body>
<wsp:getCurrentDate>
<!--Optional:-->
<arg0>?</arg0>
</wsp:getCurrentDate>
</soapenv:Body>
</soapenv:Envelope>
51949 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.core.ForwardingClient - Using information from Host header as hostname/port
51949 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.core.ForwardingClient - Got webservice forwarding request, sending to:http://localhost:9999/simple
51981 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.core.ForwardingClient - Using interceptors:[class be.error.wsproxy.interceptors.externalchain.HttpRequestHeaderTransfererInterceptor]
51981 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.core.ForwardingClient$3 - Opening [org.springframework.ws.transport.http.HttpComponentsConnection@1dd5e19a] to [http://localhost:9999/simple]
51991 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.core.ForwardingClient - Forwarding (http://localhost:9999/simple) done.
51994 [http-bio-8080-exec-5] DEBUG be.error.wsproxy.interceptors.internalchain.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate INBOUND SIDE Response:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
<S:Body>
<ns2:getCurrentDateResponse xmlns:ns2="http://wsproxy.error.be/">
<return>2013-10-28T15:55:29.717+01:00</return>
</ns2:getCurrentDateResponse>
</S:Body>
</S:Envelope>
</pre>
In the default setup default logging happens on the inbound side. The inbound interceptors are configure <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/interceptors/InboundInterceptors.java" target="_blank">here</a>:
<pre class="brush: java;">
@Configuration
public class InboundInterceptors {
@Autowired
private PayloadRootAnnotationMethodEndpointMapping catchAllEndpointMapping;
@Autowired
private MessageDispatcher messageDispatcher;
@Configuration
public static class FirstInlineInterceptors {
@Bean
public DelegatingSmartSoapEndpointInterceptor loggingInterceptor() {
return new DelegatingSmartSoapEndpointInterceptor(new LoggingInterceptor());
}
}
@Configuration
public static class ServiceSpecificInterceptors {
}
@Configuration
public static class LastInLineInterceptors {
}
}
</pre>
If you want to configure this logging interceptor also on the outbound side, you can add them in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/interceptors/OutboundInterceptors.java" target="_blank">OutboundInterceptors</a>. <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/interceptors/logging/LoggingInterceptor.java" target="_blank">LoggingInterceptor</a> both implements <i>EndpointInterceptor</i> and <i>ClientInterceptor</i>. To continue with our requirements, there is also an interceptor which is able to log fragments based on XPath expression. The <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/interceptors/logging/LoggingXPathInterceptor.java" target="_blank">LoggingXPathInterceptor</a> is service specific and hence we'll be adding this one to the ServiceSpecificInterceptors. The difference is that service specific interceptors use <a href="http://docs.spring.io/spring-ws/site/apidocs/org/springframework/ws/soap/server/endpoint/interceptor/PayloadRootSmartSoapEndpointInterceptor.html" target="_blank">PayloadRootSmartSoapEndpointInterceptor</a> which we need to give namespace and payload root element to identify the service. The configured interceptor will only be invoked for that service. The first in line and last in line use <a href="http://docs.spring.io/spring-ws/site/apidocs/org/springframework/ws/soap/server/endpoint/interceptor/DelegatingSmartSoapEndpointInterceptor.html" target="_blank">DelegatingSmartSoapEndpointInterceptor</a> which will be invoked for any request.
<pre class="brush: java;">
@Configuration
public static class ServiceSpepcificInterceptors {
@Bean
public PayloadRootSmartSoapEndpointInterceptor getCurrentDateLoggingInterecptor() {
LoggingXPathInterceptor loggingXPathInterceptor = new LoggingXPathInterceptor();
loggingXPathInterceptor.addRequestXPaths(new WebServiceMessageXPathExpressionMetaData(
"//*[local-name()='arg0']", "requestParameter"));
loggingXPathInterceptor.addResponseXPaths(new WebServiceMessageXPathExpressionMetaData(
"//*[local-name()='return']", "responseParameter"));
return new PayloadRootSmartSoapEndpointInterceptor(loggingXPathInterceptor, "http://wsproxy.error.be/",
"getCurrentDate");
}
}
</pre>
When we execute the request again in soap-ui, we can see that the request argument and response value are extracted logged to our log file:
<pre class="brush: xml;">
DEBUG be.error.wsproxy.interceptors.internalchain.LoggingXPathInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate XPATHID:requestParameter VALUE:?
DEBUG be.error.wsproxy.interceptors.internalchain.LoggingXPathInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate XPATHID:responseParameter VALUE:2013-10-28T16:50:29.537+01:00
</pre>
The <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/interceptors/logging/WebServiceMessageXPathExpressionMetaData.java" target="_blank">WebServiceMessageXPathExpressionMetaData</a> operates by default on the soap body (payload) and treats the given XPaths as mandatory (but non blocking). To see other options check the javadoc on <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/interceptors/logging/WebServiceMessageXPathExpressionMetaData.java" target="_blank">WebServiceMessageXPathExpressionMetaData</a>.<br><p>
The properties that can be configured are located in the package <i>be.error.wsproxy.configuration.properties</i>. Following classes exist:
<ul>
<li><a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/properties/EndpointProtocolMapping.java" target="_blank">EndpointProtocolMapping</a></li>
<li><a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/properties/EndpointTargetUrlMapping.java" target="_blank">EndpointTargetUrlMapping</a></li>
<li><a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/properties/ForwardProxy.java" target="_blank">ForwardProxy</a></li>
<li><a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/properties/Keystores.java" target="_blank">Keystores</a></li>
</ul>
The default Spring profile "local", enabled via Maven the default Maven filter, will resolve them from the properties file in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/config/wsproxy-local-config/wsproxy_local_demo.properties" target="_blank">wsproxy_local_demo.properties</a>. The configuration is always stored as a simple string allowing for easy externalization in for example a JNDI environment. The three first properties determine how messages will be forwarded, to start with EndpointProtocolMapping:<br><p>
In the above scenario the target URL was automatically deduced from the Host parameters as our internal client was using the module as a forward proxy. Since the host parameters do not contain any notion about protocols, the wsproxy assumed http as the forwarding protocol by default. If don't have a reverse proxy which takes care of offloading TLS, you can ask the proxy module to forward over https instead. You can do this by setting the protocol mapping to https for a specific service:
<pre class="brush: xml;">
endpoint.protocol.mapping={namespace}payloadRootElementLocalName=https,...
</pre>
EndpointTargetUrlMapping allows to define the target URL directly. This is required in the scenario where an external client will be accessing our internal service. In that case the target URL can no longer be deduced; external clients will not use our module as forward proxy but the message will simply end up on our module as it would be the actual service. The module then needs to know to where it should forward the message to:
<pre class="brush: xml;">
endpoint.target.url.mapping={namespace}payloadRootElementLocalName=http(s)://host:port/path,....
</pre>
This can also be used to override the target URL all together. The forwarder will first look if there is an explicit URL defined for the given service, if so that one will be given precedence.<br><p>
The ForwardProxy can be configured when the wsproxy module on its turn needs to communicate via a http forward proxy in order to reach the target. This is also configurable on a per service basis.
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-FjVlqqm4Lm8/Um_ftnlebGI/AAAAAAAAAaI/bJZ6XYx-2CE/s1600/wsproxy-proxy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-FjVlqqm4Lm8/Um_ftnlebGI/AAAAAAAAAaI/bJZ6XYx-2CE/s400/wsproxy-proxy.png" /></a></div>
Remember that the forward proxy does not change anything on how the target URL is calculated. Instead of directly accessing the target URL, the message will be forward to the configurered proxy if the setting is used:
<pre class="brush: xml;">
forward.proxy=={namespace}payloadRootElementLocalName=host:port,...
</pre><br>
The keystores point to the keystore configuration containing store location, store password, key alias and key password. They are used when we want to apply message security which we'll cover next.
<pre class="brush: xml;">
keystores.location=${project.root.dir}/config/test-keystores
keystore=${keystores.location}/keystore.jks
keystore.password=changeme
key.alias=mykey
key.password=changeme
truststore=${keystores.location}/truststore.jks
truststore.password=changeme
</pre>
To satisfy the last requirement (integrity/confidentiality) we'll be using WSS4J via the Spring Wss4jSecurityInterceptor. This interceptor needs to be configured on the outbound side in our example were we have an internal client accessing an external service. The steps we will perform:
<ul>
<li>Setup a secured standalone soap based endpoint</li>
<li>Configure the wsproxy with message security for the given service</li>
<li>Deploy wsproxy</li>
<li>Use a web services client to access the endpoint via the proxy module</li>
</ul>
For the secured endpoint <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/test/java/be/error/wsproxy/SimpleSecuredEndpoint.java" target="_blank">SimpleSecuredEndpoint</a> is foreseen using JAXWS and WSIT. The WSIT configuration can be found in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/test/resources/META-INF/wsit-be.error.wsproxy.SimpleSecuredEndpoint%24SimpleWebServiceEndpoint.xml" target="_blank">META-INF/wsit-be.error.wsproxy.SimpleSecuredEndpoint$SimpleWebServiceEndpoint.xml</a> enabling message integrity on our endpoint.
<pre class="brush: java;">
public class SimpleSecuredEndpoint {
public static void main(String args[]) throws IOException {
// Set WSIT_HOME manually, we're only using this for testing purposes. This way we can have a dynamic path based
// on the project location in filesystem to resolve the keystores via the WSIT configuratin in META-INF
System.setProperty("WSIT_HOME", new ClassPathResource("").getFile().getParent() + "/../config/test-keystores/");
Endpoint.publish("http://localhost:9999/simple", new SimpleWebServiceEndpoint());
}
@WebService(serviceName = "SimpleEndpoint")
@Addressing(enabled = false, required = false)
public static class SimpleWebServiceEndpoint {
public Date getCurrentDateSecured(String randomParameter) {
return new Date();
}
}
}
</pre>
<b>Important:</b> the JAXWS implementation shipped with the JDK does not contain WSIT. It is just the JAXWS RI. In order for this to work you'll have to download the latest Metro release yourself which bundles everything together. See <a href="https://metro.java.net/" target="_blank">Metro home page</a>. When you've downloaded Metro, run SimpleSecuredEndpoint passing along the endorsed system property: <b>-Djava.endorsed.dirs=/path_to_metro/lib</b>. This will ensure the entire JAXWS implementation is used from the external libraries. When everything is running fine you'll be seeing a line: <i>INFO: WSP5018: Loaded WSIT configuration from file: file:/home/koen/....</i>.<br><p>
The configuration of the WSS4J interceptor enabling message integrity in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/interceptors/OutboundInterceptors.java" target="_blank">OutboundInterceptors</a>:
<pre class="brush: java;">
@Bean
public Map<QName, List<ClientInterceptor>> customClientInterceptors() throws Exception {
Map<QName, List<ClientInterceptor>> mapping = new HashMap<>();
List<ClientInterceptor> list = new ArrayList<>();
list.add(getCurrentDateServiceSecurityInterceptor());
list.add(new LoggingInterceptor());
mapping.put(new QName("http://wsproxy.error.be/", "getCurrentDateSecured"), list);
return mapping;
}
private Wss4jSecurityInterceptor getCurrentDateServiceSecurityInterceptor() throws Exception {
Wss4jSecurityInterceptor interceptor = new Wss4jSecurityInterceptor();
// Outgoing
interceptor.setSecurementActions("Signature Timestamp");
interceptor
.setSecurementSignatureParts("{}{http://schemas.xmlsoap.org/soap/envelope/}Body;{}{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd}Timestamp");
interceptor.setSecurementSignatureKeyIdentifier("IssuerSerial");
Pair<String, String> key = keystore.getKeyAliasPasswords().get(0);
interceptor.setSecurementUsername(key.getLeft());
interceptor.setSecurementPassword(key.getRight());
interceptor.setSecurementSignatureAlgorithm("http://www.w3.org/2000/09/xmldsig#rsa-sha1");
interceptor.setSecurementSignatureDigestAlgorithm("http://www.w3.org/2000/09/xmldsig#sha1");
interceptor.setSecurementTimeToLive(700);
interceptor.setValidationTimeToLive(700);
interceptor.setSecurementSignatureCrypto(keystoreCrypto);
// Incomming
interceptor.setValidationActions("Timestamp Signature");
interceptor.setValidationSignatureCrypto(truststoreCrypto);
return interceptor;
}
</pre>
On line 6 and 7 we add the custom interceptor to the list of interceptors used by the ForwardingClient. We've also added the LoggingInterceptor on the outbound so we can see the secured messages going out and coming in. To test the message security configuration deploy the wsproxy and use soap-ui to fire the request. The soap-ui setup is no different then setup for the non-secured endpoint.<br><p>
<b>Important:</b>There seems to be an issue with C14N. When the request is sent as normal, WSIT will complain that the calculated digest does not match with the one in the message. I'm going to investigate this further, but this appears to be a problem of WSIT and not WSS4J, since the same problem also happens when soap-ui is configured as a secured client and directly communicates with the endpoint rather then using the wsproxy module. To get around this and see the test working, remove the line feed between the soap Body start element and the payload root start element. Also remove the line feed between the soap Body end element and the payload root end element:
<pre class="brush: xml; highlight: [3, 6];">
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsp="http://wsproxy.error.be/">
<soapenv:Header/>
<soapenv:Body><wsp:getCurrentDateSecured>
<!--Optional:-->
<arg0>?</arg0>
</wsp:getCurrentDateSecured></soapenv:Body>
</soapenv:Envelope>
</pre>
The soap-ui projects are also <a href="https://github.com/koen-serneels/ws-proxy/tree/master/src/test/soapui" target="_blank">availale in the project</a> if you want. <b>Don't forget to enable the proxy settings as explained before, they are not saved as part of the project.</b><br><p>
The result:
<pre class="brush: xml;">
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:exc14n="http://www.w3.org/2001/10/xml-exc-c14n#" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<S:Header/>
<S:Body wsu:Id="_5002">
<ns2:getCurrentDateSecuredResponse xmlns:ns2="http://wsproxy.error.be/">
<return>2013-10-29T14:26:25.789+01:00</return>
</ns2:getCurrentDateSecuredResponse>
</S:Body>
</S:Envelope>
</pre>
Nothing spectacular as the wsproxy added message security when forwarding the request and removed it when returning the response. If we look at the wsproxy log files, we'll first see the request entering on the inboud side:
<pre class="brush: xml;">
34 [http-bio-8080-exec-3] DEBUG be.error.wsproxy.interceptors.logging.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate INBOUND SIDE Request:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsp="http://wsproxy.error.be/">
<soapenv:Header/>
<soapenv:Body>
<wsp:getCurrentDateSecured>
<!--Optional:-->
<arg0>?</arg0>
</wsp:getCurrentDateSecured>
</soapenv:Body>
</soapenv:Envelope>
</pre>
The request is secured and forward to the endpoint:
<pre class="brush: xml;">
394 [http-bio-8080-exec-3] DEBUG be.error.wsproxy.interceptors.logging.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate OUTBOUND SIDE Request:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsp="http://wsproxy.error.be/">
<soapenv:Header>
<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" soapenv:mustUnderstand="1">
<wsu:Timestamp wsu:Id="TS-518848887F924441AB13830540361321">
<wsu:Created>2013-10-29T13:40:36.130Z</wsu:Created>
<wsu:Expires>2013-10-29T13:45:36.130Z</wsu:Expires>
</wsu:Timestamp>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#" Id="SIG-518848887F924441AB13830540361916">
<ds:SignedInfo>
...
</pre>
The secured response is received from the endpoint;
<pre class="brush: xml;">
524 [http-bio-8080-exec-3] DEBUG be.error.wsproxy.interceptors.logging.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate OUTBOUND SIDE Response:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:exc14n="http://www.w3.org/2001/10/xml-exc-c14n#" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<S:Header>
<wsse:Security S:mustUnderstand="1">
<wsu:Timestamp xmlns:ns15="http://www.w3.org/2003/05/soap-envelope" xmlns:ns16="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" wsu:Id="_3">
<wsu:Created>2013-10-29T13:40:36Z</wsu:Created>
<wsu:Expires>2013-10-29T13:45:36Z</wsu:Expires>
</wsu:Timestamp>
<ds:Signature xmlns:ns15="http://www.w3.org/2003/05/soap-envelope" xmlns:ns16="http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512" Id="_1">
<ds:SignedInfo>
...
</pre>
Security information is processed and validated. If ok, the security information is stripped and the response returned (to our client, soap-ui in this case):
<pre class="brush: xml;">
567 [http-bio-8080-exec-3] DEBUG be.error.wsproxy.interceptors.logging.LoggingInterceptor - SID:{http://wsproxy.error.be/}getCurrentDate INBOUND SIDE Response:<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:exc14n="http://www.w3.org/2001/10/xml-exc-c14n#" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<S:Header/>
<S:Body wsu:Id="_5002">
<ns2:getCurrentDateSecuredResponse xmlns:ns2="http://wsproxy.error.be/">
<return>2013-10-29T14:40:36.357+01:00</return>
</ns2:getCurrentDateSecuredResponse>
</S:Body>
</S:Envelope>
</pre>
Until now all tests have been done simulating an internal client accessing an external service. In case you want to use the module to do the inverse; serving external clients accessing internally hosted services, it is more of the same. For each internally hosted service to which the module will forward you would have to register a target URL using the <i>endpoint.target.url.mapping</i> configuration parameter. Interceptors continue working the same way, but remember that for example for message security you probably want the Wss4jSecurityInterceptor to be configured on the inbound side, as in this scenario the inbound side is the external facing side. There is no issue in configuring the Wss4jSecurityInterceptor on both inbound and outbound side for different service; all configuration is based on a per service basis.<br><p>
For example: service x (identified by namespace and payload root element) is an internal hosted service. Service y is an external service which internal clients want to access. For securing our internal service x we would be adding Wss4jSecurityInterceptor as service specific inbound interceptor in the <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/interceptors/InboundInterceptors.java" target="_blank">InboundInterceptors</a> configuration. This interceptor will thus only be active on the wsproxy endpoint (only serving the inbound side, in this example the external facing side) and only for service x. For securing the calls to service y, we would register Wss4jSecurityInterceptor in <a href="https://github.com/koen-serneels/ws-proxy/blob/master/src/main/java/be/error/wsproxy/configuration/interceptors/OutboundInterceptors.java" target="_blank">OutboundInterceptors</a>, adding message security for messages being sent to the external service y by the wsproxy module.<br><p>
Ok, that's about it! Feel free to drop me a message if this was somehow useful or if you have ideas for improvement!Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com2tag:blogger.com,1999:blog-3406746278542965235.post-5794388962620370312013-09-08T15:26:00.000+02:002013-09-09T11:29:28.577+02:00WS-Security: using BinarySecurityToken for authenticationAs we all know, one goal set by WS-Security is to enforce integrity and/or confidentially on SOAP messages. In case of integrity, the signature which is added to the SOAP message is the result of a mathematical process involving the private key of the sender resulting in an encrypted message digest.<p>
Most frameworks, such as WSS4J, will by default only sign the body. If you're adding extra headers, such as a Timestamp header, you'll have indicate explicitly to sign them. Using the Spring support for WSS4J for example, you can set a comma separated list containing the local element name and the corresponding namespace using the <i>securementSignatureParts</i> property.<p>
Below an example how to instruct it to both sign the Body and Timestamp element (and their siblings). This will result in two digital signatures being appended to the message:
<pre class="brush: xml;">
<property name="securementSignatureParts" value="{}{http://schemas.xmlsoap.org/soap/envelope/}Body;{}{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd}Timestamp" />
</pre>
Eventually the SOAP message will be send together with the XML digital signature data and in most cases a BinarySecurityToken containing the certificate.<p>
Nothing new so far. However, what struck me is that it seems not widely understood what the goal is of the BST neither how authentication is controlled using it. Let me try to shed some light on this:<p>
The certificate of the sender which is send along with the SOAP message plays the role of identification. You can compare it as being the username++. It should be clear that the certificate inside the message cannot be trusted, neither can a username without verifying the password. So far everyone agrees on that: <i>"yeah of course, certificates need to be validated in order to be trusted and then you're set!"</i><p>
But that is not the entire story. Validation of the certificate is not the same as authentication. The fact that the certificate in the message is valid and is signed by a known CA is not enough to consider the sender authenticated.<p>
For example: I, in my most malicious hour, could have intercepted the message, changed the content, created a new signature based on my private key and replaced the BST in the message with my certificate. My certificate could perfectly be an official CA signed certificate (even signed by the same CA as you're using) so it would pass the validation check. If the framework would simply validate the certificate inside the message we would have no security at all.<p>
<i>Note: If you're sending the message over secure transport instead, chances are that I was not able to intercept the message.
But secure transport is mostly terminated before the actual endpoint, leaving a small piece of the transport "unsecured".
Albeit this part will be mostly internally in your company, but what I want to point out is that no matter how secure your transport is, the endpoint has the end responsibility in verifying the identity of the sender. For example; in an asynchronous system the SOAP message could have been placed on a message queue to be processed later.
When processing starts by the endpoint, the trace of the secure transport is long gone. You'll have to verify the identity using the information contained in the message.</i><p>
In order to close this loophole we have two solutions:<p>
The first solution builds further on what we already described: the certificate in the message is verified against the CA root certificates in the truststore. In this scenario it advised to first narrow the set of trusted CA's. You could for example agree with your clients on a limited list of CA's to get your certificates from. Doing so you are already lowered the risk of trusting more "gray zone" CA's which might not take the rules for handing out certificates so strict (like for example, proper checking the identity of their clients). Secondly, because <b>*every*</b> certificate handed out by your trusted CA will be considered "authenticated", we'll close the loophole by issuing some extra checks.<p>
Using WSS4J you can configure a matching pattern based on the subject DN property of the certificate. They have a nice blog entry on this here: <a href="http://coheigea.blogspot.ie/2012/08/subject-dn-certificate-constraint.html" target="_blank">http://coheigea.blogspot.ie/2012/08/subject-dn-certificate-constraint.html</a>.<br>
We could specify that the DN of the certificate must match a given value like this:
<pre class="brush: java;">
Wss4jHandler handler = ...
handler.setOption(WSHandlerConstants.SIG_SUBJECT_CERT_CONSTRAINTS, "CN = ...");
</pre>
Note: that there is currently no setter for this using the Spring support for WSS4J in Wss4jSecurityInterceptor, so you'll have to extend it in order to enable this!<p>
To conclude the steps being performed:
<ol>
<li>The certificate contained in the message is validated against the trusted CA in your trustore. When this validation succeeds it tells the application that the certificate is still valid and has actually been handed out by a CA that you consider trusted.
<ul><li>
This check gives us the guarantee that the certificate really belongs to the party that the certificate claimes to belong to.
</li></ul>
</li>
<li>Optionally the certificate can also be checked on revocation so that we don't continue trusting certificates that are explictly been revoked.</li>
<li>WSS4J will check if some attributes of the certificate match the required values for the specific service (Subject DN Certificate Constraint support).
<ul><li>This would be the authentication step; once the certficate has been found valid, we check if the owner of the certificate is the one we want to give access too</li></ul>
</li>
<li>Finally the signature in the message is verified by creating a new digest of the message, compare it with decrypted digest from the message and so forth</li>
</ol>
It should be noted that this check (at least when using WSS4J) is not done by default! If you don't specify it and simply add your CA's in the trust store you'll be leaving a security hole!<p>
The second solution requires no extra configuration and depends on ONLY the certificate of the sender to be present in the truststore.<br>
The certificate contained in the message is matched against the certificate in the truststore. If they match the sender is authenticated. There is no need to validate certificates against a CA since the certificates imported in the truststore are explicitly trusted (WSS4J will still check if the certificate is not expired and possibly check it for revocation). Again, there are no CA certificates (or CA intermediate certificates) in the truststore! Only the certificates of the senders that you want to give access too. Access is hereby controlled by adding (or removing) their certificate from the truststore.<p>
This requires you to be cautious when initially importing the certificates since you'll have to make sure they actually represent the sender.
But this is something you're always obliged to do when adding certificates to your truststore, also when adding CA certificates like in the first solution.<p>
<b>Conclusion:</b> in the assumption you can limit the trusted CA's, the first solution is in most cases the preferred one and also the most scalable. For new clients there are no changes required to the truststore. The attributes to match can be stored externally so they are easy to change/add. Also, when the client certificates expires or gets revoked, you don't need to do anything special. The new certificate will we used by the sender at a given moment and will directly be validated against the CA in your truststore. In the second solution you would have to add the new certificate to the trustore and leave the old one in there for a while until the switch is performed.<p>
Overall lessons learned: water tight security is hard. The #1 rule in IT (assumption is the mother of all f***ups) is certainly true here.
Be skeptical and make sure you fully understand what is going on. Never trust default settings until you are sure what they do.
The default setting on your house alarm (eg. 123456) is no good idea either. Neither is the default admin password on a Tomcat installation.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-55532470580836537002013-07-26T19:30:00.002+02:002013-07-26T19:30:44.187+02:00Splitting Large XML Files in JavaLast week I was asked to write something in Java that is able to split a single 30GB XML file into smaller parts of configurable file size. The consumer of the file is going to be a middle-ware application that has problems with the large size of the XML. Under the hood it uses some kind of DOM parsing technique that causes it to run out of memory after a while. Since it's vendor based middle-ware, we are not able to correct this ourselves. Our best option is to create some pre-processing tool that will first split the big file in multiple smaller chunks before they are processed by the middle-ware.<p>
The XML file comes with a corresponding W3C schema, consisting of a mandatory header part followed by a content element which has several 0..* data elements nested. For the demo code I re-created the schema in simplified form:
<table><tr><td><a href="http://2.bp.blogspot.com/-oT_98sBWMcs/Ue_6mMsw6wI/AAAAAAAAAV8/s_JThe2DHQk/s1600/schema.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-oT_98sBWMcs/Ue_6mMsw6wI/AAAAAAAAAV8/s_JThe2DHQk/s320/schema.png" /></a></td><td><a href="http://1.bp.blogspot.com/-UhLjC_1Y2Xc/UfAAZY3cAMI/AAAAAAAAAWQ/QPbG-yFFxuk/s1600/schema.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-UhLjC_1Y2Xc/UfAAZY3cAMI/AAAAAAAAAWQ/QPbG-yFFxuk/s320/schema.png" /></a></td></tr></table>
The header is neglectable in size. A single data element repetition is also pretty small, lets say less less then 50kB. The XML is so big because of the number of repetitions of the data element. The requirements are that:
<ul>
<li>Each part of the splitted XML should be syntactical valid XML and each part should also validate against the original schema</li>
<li>The tool should validate the XML against the schema and report any validation errors. Validation must not be blocking and non validating elements or attributes must not be skipped in the output</li>
<li>For the header there is decided that rather then copying it in each of the new output files, the header will be re-generated for each new output file with some information of the processing and some defaults</li>
</ul>
So, using binary split tools such as Unix Split is out of the question. This will split after a fixed number of bytes leaving the XML corrupt for sure. I'm not really sure but tools such as Split don't know anything about encoding either. So splitting after byte 'x' could not only result in splitting in the middle of an XML element (for example), but even in the middle of a character encoding sequence (when using Unicode that is UTF8 encoded for example). It's clear we need something more intelligent.<p>
XSLT as core technology is no go either. At first sight one could be tempted: using XSLT2.0 it is possible to create multiple output files from a single input file. It should even be possible to validate the input file while transforming. However, the devil is, as always, in the details. Otherwise simple operations in Java such as writing the validation error's to a separate file, or checking the size of the current output file would probably require custom java code. This is certainly possible with Xalan and Saxon to have such extensions, but Xalan is not a XSLT2.0 implementation so that only leaves us with Saxon. Last but not least, XSLT1.0/2.0 are non-streaming, meaning that they will read the entire source document into memory, so this clearly excludes XSLT from the possibilities.<p>
This leaves us with Java XML parsing as the only option left. The ideal candidate is in this case of course StAX. I'm not going into the SAX ↔ StAX comparison here, fact is that StAX is able to validate against schema's (at least some parsers can) and can also write XML. Further more, the API's is a lot easier to use then SAX, because it pull based it gives more control on iterating the document and works more pleasant then the push way of SAX. Aight, what do we need:
<ul>
<li>StAX implementation capable of validating XML
<ul><li>
Oracle's JDK ships by default with SJSXP as the StAX implementation, but this one is however non validating; so I ended up with using Woodstox. As far as I could find, validation with Woodstox is only possible using the StAX cursor API</li></ul></li>
<li>Preferably have some Object/XML mapping technique for (re)creating the header instead of manually fiddling with elements and having to look up the correct datatypes/format
<ul><li>Clearly JAXB. It has support for StAX, so you can create your Object model and then let it directly write to a StAX outputstream</li></ul></li></ul>
The code is a bit to large to show it here as a whole. Both the source files, XSD and test XML can be accessed <a target="_blank" href="https://github.com/koen-serneels/blog/tree/master/BigXmlTest/src/main/java/be/error/bigxmltest">here</a> on GitHub. It has a Maven pom file so you should be able to import it in your IDE of choice. The JAXB binding compiler will automatically compile the schema and put the generated sources on the classpath.<p>
<pre class="brush: java;">
public void startSplitting() throws Exception {
XMLStreamReader2 xmlStreamReader = ((XMLInputFactory2) XMLInputFactory.newInstance())
.createXMLStreamReader(BigXmlTest.class.getResource("/BigXmlTest.xml"));
PrintWriter validationResults = enableValidationHandling(xmlStreamReader);
int fileNumber = 0;
int dataRepetitions = 0;
XMLStreamWriter xmlStreamWriter = openOutputFileAndWriteHeader(++fileNumber); // Prepare first file
</pre>
The first line creates our StAX stream reader which means we are using the cursor API. The iterator API uses the XMLEventReader class. There is also a strange "2" in the classname which refers to the StAX 2 features from Woodstox, one of them is probably the support for validation. From <a target="_blank" href="http://docs.codehaus.org/display/WSTX/StAX2">here</a>:
<pre>StAX2 is an experimental API that is intended to extend basic StAX specifications
in a way that allows implementations to experiment with features before they
end up in the actual StAX specification (if they do). As such, it is intended
to be freely implementable by all StAX implementations same way as StAX, but
without going through a formal JCP process. Currently Woodstox is the only
known implementation.</pre>
"enableValidationHandling" can be seen in the <a target="_blank" href="https://github.com/koen-serneels/blog/blob/master/BigXmlTest/src/main/java/be/error/bigxmltest/BigXmlTestCursorApi.java">source file</a> if you want. I'll highlight the important pieces. First, load the XML schema:
<pre class="brush: java;">
XMLValidationSchema xmlValidationSchema = xmlValidationSchemaFactory.createSchema(BigXmlTest.class
.getResource("/BigXmlTest.xsd"));
</pre>
Callback for writing possible validation results to the output file;
<pre class="brush: java;">
public void reportProblem(XMLValidationProblem validationError) throws XMLValidationException {
validationResults.write(validationError.getMessage()
+ "Location:"
+ ToStringBuilder.reflectionToString(validationError.getLocation(),
ToStringStyle.SHORT_PREFIX_STYLE) + "\r\n");
}
</pre>
The "openOutputFileAndWriteHeader" will create a XMLStreamWriter (which is again part form the cursor API, the iterator API has XMLEventWriter) to which we can output or part of the original XML file. It will also use JAXB to create our header and let it write to the output. The JAXB objects are generated default by using the Schema compiler (xjc).
<pre class="brush: java;">
private XMLStreamWriter openOutputFileAndWriteHeader(int fileNumber) throws Exception {
XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newInstance();
xmlOutputFactory.setProperty(XMLOutputFactory.IS_REPAIRING_NAMESPACES, true);
XMLStreamWriter writer = xmlOutputFactory.createXMLStreamWriter(new FileOutputStream(new File(System
.getProperty("java.io.tmpdir"), "BigXmlTest." + fileNumber + ".xml")));
writer.setDefaultNamespace(DOCUMENT_NS);
writer.writeStartDocument();
writer.writeStartElement(DOCUMENT_NS, BIGXMLTEST_ROOT_ELEMENT);
writer.writeDefaultNamespace(DOCUMENT_NS);
HeaderType header = objectFactory.createHeaderType();
header.setSomeHeaderElement("Something something darkside");
marshaller.marshal(new JAXBElement<HeaderType>(new QName(DOCUMENT_NS, HEADER_ELEMENT, ""), HeaderType.class,
HeaderType.class, header), writer);
writer.writeStartElement(CONTENT_ELEMENT);
return writer;
}
</pre>
On line 3 we enable "repairing namespaces". The specification has this to say:
<pre>
javax.xml.stream.isRepairingNamespaces:
Function: Creates default prefixes and associates them with Namespace URIs.
Type: Boolean
Default Value: False
Required: Yes
</pre>
What I understand from this is that its required for handling default namespaces. Fact is that if it is not enabled the default namespace is not written in any way. On line 6 we set the default namespace. Setting it does not actually write it to the stream. Therefore one needs writeDefaultNamespace (line 9) but that can only be done after a start element has been written. So, you have to define the default namespace before writing any elements, but you need to write the default namespace after writing the first element. The rationale is that StAX needs to know if it has to generate a prefix for the root element you are going to write yes or no.<p>
On line 8 we write the root element. Important to indicate to which namespace this element belongs. If you do not specify a prefix, a prefix will be generated for you, or, in our case no prefix will be generated at all because StAX knows we already set the default namespace. If you would remove the indication of the default namespace at line 6, the root element will be prefixed (with a random prefix) like: <wstxns1:BigXmlTest xmlns:wstxns1="http://www... Next we write our default namespace, this will be written to the element started previously (btw, for some deeper understanding on this order see this nice <a target="_blank" href="http://veithen.blogspot.co.uk/2009/10/understanding-stax-how-to-correctly-use.html">article</a>). On line 11-14 we use our JAXB generated model to create the header and let our JAXB marshaller write it directly to our StAX output stream.<p>
<b>Important:</b> the JAXB marshaller is initialized in fragment mode, otherwise it will start to add an XML declaration, as would be required for standalone documents, and that is of course not allowed in the middle of an existing document:
<pre class="brush: java;">
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
</pre>
On a side note: the JAXB integration is not really useful in this example, it creates more complexity and takes more lines of code then simply adding the elements using the XMLStreamWriter. However, in if you have a more complex structure which you need to create and merge into the document it is pretty handy to have automatic object mapping.<p>
So, we have our reader which is enabled for validation. From the moment we start iterating over the source document it will validate and parse at the same time. Then we have our writer which already has an initialized document and header written and is ready to accept more data. Finally we have to iterate over the source and write each part to the output file. If the output file becomes to big we will switch it with a new one:
<pre class="brush: java;">
while (xmlStreamReader.hasNext()) {
xmlStreamReader.next();
if (xmlStreamReader.getEventType() == XMLEvent.START_ELEMENT
&& xmlStreamReader.getLocalName().equals(DATA_ELEMENT)) {
if (dataRepetitions != 0 && dataRepetitions % 2 == 0) { // %2 = just for testing: replace this by for example checking the actual size of the current output file
xmlStreamWriter.close(); // Also closes any open Element(s) and the document
xmlStreamWriter = openOutputFileAndWriteHeader(++fileNumber); // Continue with next file
dataRepetitions = 0;
}
// Transform the input stream at current position to the output stream
transformer.transform(new StAXSource(xmlStreamReader), new StAXResult(
new FragmentXMLStreamWriterWrapper(new AvoidDefaultNsPrefixStreamWriterWrapper(xmlStreamWriter, DOCUMENT_NS))));
dataRepetitions++;
}
}
</pre>
The important bits are that we keep iterating over the source document and check for the presence of the start of the Data element. If so we stream the corresponding element and its siblings to the output. In our simple example we don't have siblings, just a text value. But if the structure is more complex all underlying nodes will automatically be copied to the output. Every two Data elements we will cycle our output file. The writer is closed and a new one is initialized (this check can of course be replaced by checking file size instead of % 2). If the writer is closed it will automatically take care of closing open elements and finally closing the document itself, no need to do this yourself. As what the mechanism is concerned for streaming the nodes from the input to the output:
<ul>
<li>Because we are forced to use the cursor API because of the validation, we have to use XSLT to transfer the node and its siblings to the output. XSLT has some default templates which will be invoked if you do not specify an XSL specifically. In this case it will transform the input to the given output.</li>
<li>A custom <a target="_blank" href="https://github.com/koen-serneels/blog/blob/master/BigXmlTest/src/main/java/be/error/bigxmltest/FragmentXMLStreamWriterWrapper.java">FragmentXMLStreamWriterWrapper</a> is needed, I documented this in the JavaDoc. This wrapper is again wrapped in a <a target="_blank" href="https://github.com/koen-serneels/blog/blob/master/BigXmlTest/src/main/java/be/error/bigxmltest/AvoidDefaultNsPrefixStreamWriterWrapper.java">AvoidDefaultNsPrefixStreamWriterWrapper</a>. The reason for this last one is that the default XSLT template does not recognize the default namespace in our source document. More on that in a minute (or search for AvoidDefaultNsPrefixStreamWriterWrapper).
</li>
<li>The transformer that you use needs to be the Oracle JDK's internal version. Where we initialize the transformer we directly reference the instance of the internal TransformerFactory: <b>com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl</b> which then creates the correct transformer: <i>transformer = new TransformerFactoryImpl().newTransformer();</i> Normally you would be using the TransformerFactory.newInstance() and use the transformer available on the classpath. However, parsers and transformers can install themselves by supplying META-INF/services. If another transformer (such as default Xalan, not the repackaged JDK version) would be on the classpath the transformation would fail. The reason is that apparently only the JDK internal version has the ability to transform from StAXSource to StAXResult</li>
<li>The transformer will actually let our XMLStreamReader continue in the iteration process. So after a Data element has been processed, the cursor of the reader will be in theory ready at the next Data element. In theory that is, since the next event type might be a space if your XML is formatted. So it still might need some iterations on the xmlStreamReader.next() in our while loop before the next Data element is actually ready.
</ul>
The result is that we have 3 output files, each compliant to the original schema, that have each 2 data elements:
<table><tr><td><a href="http://1.bp.blogspot.com/-pBdka-zVYqI/UfAncrBveBI/AAAAAAAAAXM/X1eEWRp6AqU/s1600/files.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-pBdka-zVYqI/UfAncrBveBI/AAAAAAAAAXM/X1eEWRp6AqU/s320/files.png" /></a></td><td><a href="http://2.bp.blogspot.com/-phA0899nVl4/UfAndcsVnPI/AAAAAAAAAXU/nQ04aK_yalM/s1600/file.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-phA0899nVl4/UfAndcsVnPI/AAAAAAAAAXU/nQ04aK_yalM/s320/file.png" /></a></td></tr></table><p>
To split a ~30GB XML (I'm talking about my original assignment XML with a more complex structure, not the demo XSD used here) in parts of ~500MB with validation it took about 25 minutes. To test the memory usage I deliberately set the Xmx to 32MB. As you can see in the graph memory consumption is very low and there is no GV overhead:<p>
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-l63hFOUgV64/UfD26tzZQOI/AAAAAAAAAXk/ldaN9F_3nF4/s1600/bigxmltest-vm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-l63hFOUgV64/UfD26tzZQOI/AAAAAAAAAXk/ldaN9F_3nF4/s400/bigxmltest-vm.png" /></a></div><p>
Life is good, but not completely. There where some awkward things that I discovered that one needs to be careful about.<p>
In my real scenario the input XML had no namespaces associated with it and I'm pretty sure it never will. That is the reason I stick with this solution. In the demo here there is a single namespace and that already starts to make the setup more brittle. The issue is not StAX: handling the namespaces with StAX is pretty simple. You can decide to have a default namespace (assuming your Schema is elementFormDefault = qualified) corresponding with the target namespace of the Schema and maybe declare some prefixed namespaces for possibly other namespaces that are imported in the Schema. The problems begin (as you might already noticed by now) when XSLT starts to interfere with the output stream. Apparently it doesn't check which namespaces are already defined or other things happen.<p>
The result is that they seriously clutter up the document by re-defining existing namespaces with other prefixes or resetting the default namespace and other stuff you don't want. One probably needs an XSL if you need more namespace manipulation then the default template. XSLT will also trigger exceptions if the input document is using default namespaces. It will try to register a prefix with name "xmlns". This is not allowed as xmlns is reserved for indicating the default namespace it cannot be used as a prefix. The fix I applied for this test was to ignore any prefix that is "xmlns" and to ignore the addition of the target namespace in cominbation with the xmlns prefix (thats why we have the AvoidDefaultNsPrefixStreamWriterWrapper). The prefix and namespace both need to match in the AvoidDefaultNsPrefixStreamWriterWrapper, because if you would have an input document without default namespace but with prefixes instead (like <bigxml:BigXmlTest xmlns:bigxml="http://...."><bigxml:Header....) then you cannot ignore adding the namespace (the combination will then be the target namespace with the "bigxml" prefix) since that would yield only prefixes for the data elements without namespaces being bound to them, for example:
<pre class="brush: java; highlight: [7,8]">
<?xml version='1.0' encoding='UTF-8'?>
<BigXmlTest xmlns="http://www.error.be/bigxmltest">
<Header>
<SomeHeaderElement>Something something darkside</SomeHeaderElement>
</Header>
<Content>
<bigxml:Data>Data1</bigxml:Data>
<bigxml:Data>Data2</bigxml:Data>
</Content>
</BigXmlTest>
</pre>
Remember that the producer of the XML is free (again in case elementFormDefault = qualified) to choose whether to use the defaultnamespace or to prefix every element. The code should transparently be able to deal with both scenario's. The AvoidDefaultNsPrefixStreamWriterWrapper code for convenience:
<pre class="brush: java;">
public class AvoidDefaultNsPrefixStreamWriterWrapper extends XMLStreamWriterAdapter {
...
@Override
public void writeNamespace(String prefix, String namespaceURI) throws XMLStreamException {
if (defaultNs.equals(namespaceURI) && "xmlns".equals(prefix)) {
return;
}
super.writeNamespace(prefix, namespaceURI);
}
@Override
public void setPrefix(String prefix, String uri) throws XMLStreamException {
if (prefix.equals("xmlns")) {
return;
}
super.setPrefix(prefix, uri);
}
</pre>
Finally, I also wrote a version (click <a target="_blank" href="https://github.com/koen-serneels/blog/blob/master/BigXmlTest/src/main/java/be/error/bigxmltest/BigXmlTestIteratorApi.java">here</a> for GitHub) that does the exact same thing but this time with the StAX iterator API. You will notice that there no longer the cumbersome XSLT required for streaming to the output. Each Event of interest is simply added to the output. The lack of validation could be solved by first validating the input using the cursor API and then parse it using the Iterator API. It will take longer, but that might still be acceptable in most conditions. The most important piece:
<pre class="brush: java; highlight: [2,4,19]">
while (xmlEventReader.hasNext()) {
XMLEvent event = xmlEventReader.nextEvent();
if (event.isStartElement() && event.asStartElement().getName().getLocalPart().equals(CONTENT_ELEMENT)) {
event = xmlEventReader.nextEvent();
while (!(event.isEndElement() && event.asEndElement().getName().getLocalPart()
.equals(CONTENT_ELEMENT))) {
if (dataRepetitions != 0 && event.isStartElement()
&& event.asStartElement().getName().getLocalPart().equals(DATA_ELEMENT)
&& dataRepetitions % 2 == 0) { // %2 = just for testing: replace this by for example checking the actual size of the current
// output file
xmlEventWriter.close(); // Also closes any open Element(s) and the document
xmlEventWriter = openOutputFileAndWriteHeader(++fileNumber); // Continue with next file
dataRepetitions = 0;
}
// Write the current event to output
xmlEventWriter.add(event);
event = xmlEventReader.nextEvent();
if (event.isEndElement() && event.asEndElement().getName().getLocalPart().equals(DATA_ELEMENT)) {
dataRepetitions++;
}
}
}
}
</pre>
On line 2 you will see that we get an XMLEvent back which contains all information about the current node. On line 4 you see that it is easier to use this form to check for the element type (instead of comparing with constants you can use the object model). On line 19 to copy the element from input to output we simply add the Event to the XMLEventWriter.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com9tag:blogger.com,1999:blog-3406746278542965235.post-5282219755596893602013-04-01T13:22:00.000+02:002013-04-01T14:54:58.891+02:00JMS and Spring: Small Things Sometimes Matter<a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/core/JmsTemplate.html">JmsTemplate</a> and <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/listener/DefaultMessageListenerContainer.html">DefaultMessageListenerContainer</a> are Spring helpers for accessing JMS compatible MOM. Their main goal is to form a layer above the JMS API and deal with infrastructure such as transaction management/message acknowledgement and hiding some of the repetitive and clumsy parts of the JMS API (hang in there: <a target="_blank" href="http://java.net/projects/jms-spec/pages/Home">JMS 2.0 </a> is on its way!). To use either one of these helpers you have to supply them with (at least) a <a target="_blank" href="http://docs.oracle.com/javaee/6/api/javax/jms/ConnectionFactory.html">JMS ConnectionFactory</a> and a valid <a target="_blank" href="http://docs.oracle.com/javaee/6/api/javax/jms/Destination.html">JMS Destination</a>.<p>
When running your app on an application server, the ConnectionFactory will most likely be defined using the JEE architecture. This boils down adding the ConnectionFactory and its configuration parameters allowing them to be published in the directory service under a given alias (eg. jms/myConnectionFactory). Within your app you might for example use the "jndi-lookup" out of the JEE namespace or JndiTemplate/JndiObjectFactoryBean beans if more configuration is required for looking up the ConnectionFactory and pass it along to your JmsTemplate and/or DefaultMessageListenerContainer.<p>
The latter, JMS destination, identifies a JMS Queue or Topic for which you want to produce messages to or consume mesages from. However, both JmsTemplate as DefaultMessageListenerContainer have two different properties for injecting the destination. There is a method taking the destination as <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/listener/AbstractMessageListenerContainer.html#setDestinationName%28java.lang.String%29">String</a> and one taking it as a <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/listener/AbstractMessageListenerContainer.html#setDestination%28javax.jms.Destination%29">JMS Destination</a> type. This functionality is nothing invented by Spring, the <a target="_blank" href="http://www.oracle.com/technetwork/java/docs-136352.html">JMS specification</a> mentions both approaches:
<pre class="brush: plain;">
4.4.4 Creating Destination Objects
Most clients will use Destinations that are JMS administered objects that they have looked up via JNDI. This is the most portable approach.
Some specialized clients may need to create Destinations by dynamically manufacturing one using a provider-specific destination name.
Sessions provide a JMS provider-specific method for doing this.
</pre>
If you pass along a destination as String then the helpers will hide the extra steps required to map them to a valid JMS Destination. In the end a <a target="_blank" href="http://docs.oracle.com/javaee/6/api/javax/jms/Session.html#createConsumer%28javax.jms.Destination%29">createConsumer on a JMS Session</a> expects you to pass along a Destination object to indicate where to consume messages from before returning a <a target="_blank" href="http://docs.oracle.com/javaee/6/api/javax/jms/MessageConsumer.html">MessageConsumer</a>.<p>
When destinations are configured as String, the Destination is looked up by Spring using the JMS API itself. By default JmsTemplate and DefaultMessageListenerContainer have a reference to a <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/support/destination/DestinationResolver.html">DestinationResolver</a> which is <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/support/destination/DynamicDestinationResolver.html">DynamicDestinationResolver</a> by default (more on that later). The code below is an extract from DynamicDestinationResolver, the highlighted lines indicate the usage of the JMS API to transform the String to a Destination (in this example a Queue):
<pre class="brush: java; highlight: [4,8]">
protected Queue resolveQueue(Session session, String queueName) throws JMSException {
if (session instanceof QueueSession) {
// Cast to QueueSession: will work on both JMS 1.1 and 1.0.2
return ((QueueSession) session).createQueue(queueName);
}
else {
// Fall back to generic JMS Session: will only work on JMS 1.1
return session.createQueue(queueName);
}
}
</pre>
The other way mentioned by the spec (JNDI approach) is to configure Destinations as administrable objects on your application server. This follows the principle as with the ConnectionFactory; the destination is published in the applications servers directory and can be looked up by its JNDI name (eg. jms/myQueue). Again you can lookup the JMS Destination in your app and pass it along to JmsTemplate and/or DefaultMessageListenerContainer making use of the property taking the JMS Destination as parameter.<p>
Now, why do we have those two options? <p>
I always assumed that it was a matter of choice between convenience (dynamic approach) and environment transparancy/configurability (JNDI approach). For example: in some situations the name of the physical destination might be different depending on the environment where your application runs. If you configure your physical destination names inside your application you obviously loose this benefit as they cannot be altered without rebuilding your application. If you configured them as administered object on the other hand, it is merely a simple change in the application server configuration to alter the physical destination name.<p>
Remember; having physical Destinations names configurable can make sense. Besides the Destination type, applications dealing with messaging are agnostic to its details. A messaging destination has no functional contract and none of its properties (physical destination, persistence, and so forth) are of importance for the code your write. The actual contract is inside the messages itself (the headers and the body). A database table on the other is an example of something that does expose a contract by itself and is tightly coupled with your code. In most cases renaming a database table does impact your code, hence making something like this configurable has normally no added value compared to a messaging Destination.<p>
Recently I discovered that my understanding of this is not the entire truth. The specification (from "4.4.4 Creating Destination Objects" as pasted some paragraphs above) already gives a hint: "Most clients will use Destinations that are JMS administered objects that they have looked up via JNDI. This is the most portable approach." Basically this tells us that the other approach (the dynamic approach where we work with a destination as String) is "the least portable" way. This was never really clear to me as each provider is required to implement both methods, however "portable" has to be looked at in a broader context.<p>
When configuring Destinations as String, Spring will by default transform them to JMS Desintations whenever it creates a new JMS Session. When using the DefaultMessageListenerContainer for consuming messages each message you process occurs in a transaction and by default the JMS session and consumer are not pooled, hence they are re-created for each receive operation. This results in transforming the String to a JMS Destination each time the container checks for new messages and/or receives a new message. The "non portable" aspect comes into play as it also means that the details and costs of this transformation depend entirely on your MOM's driver/implementation. In our case we experienced this with Oracle AQ as MOM provider. Each time a destination transformation happens the driver executes a specific query:
<pre class="brush: sql">
select /*+ FIRST_ROWS */ t1.owner, t1.name, t1.queue_table, t1.queue_type, t1.max_retries, t1.retry_delay, t1.retention, t1.user_comment, t2. type , t2.object_type, t2.secure
from all_queues t1, all_queue_tables t2
where t1.owner=:1 and t1.name=:2 and t2.owner=:3 and t1.queue_table=t2.queue_table
</pre>
Forum entry <a target="_blank" href="https://forums.oracle.com/forums/thread.jspa?threadID=1097902&tstart=4">can be found here</a>.<p>
Although this query was improved in the latest drivers (as mentioned by the bug report), it was still causing significant overhead on the database. The two options to solve this:<p>
<ul>
<li>Do what the specification advices you to do: configure destinations as resources on the application server. The application server will hand out the same instance each time, so they are already cached there.
Even though you will receive the same instance for every lookup, when using JndiTemplate (or JndiDestinationResolver, see below) it will also be chached application side, so even the lookup itself will only happen once.</li>
<li>Enable session/consumer caching on the DefaultMessageListenerContainer. When the caching is set to consumer, it indirectly also re-use the Destination as the consumer holds a reference to the Destination.
This pooling is Spring added functionality and the <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/listener/DefaultMessageListenerContainer.html#setCacheLevel%28int%29">JavaDoc</a> says it safe when using resource local transaction and it "should" be safe when using XA transaction (except running on JBoss 4).</li>
</ul>
The first is probably the best. However in our case all destinations are already defined inside the application (and there are plenty of them) and there is no need for having them configurable. Refactoring these merely for this technical reason is going to generate a lot of overhead with no other advantages. The second solution is the least preferred one as this would imply extra testing and investigation to make sure nothing breaks. Also, this seems to be doing more then needed, as there is no indication in our case that creating a Session or Consumer has measurable impact on performance. According to the JMS specification:
<pre class="brush: plain;highlight: [4]">
4.4 Session
A JMS Session is a single-threaded context* for producing and consuming
messages. Although it may allocate provider resources outside the Java virtual
machine, it is considered a lightweight JMS object.
</pre>
Btw; this is also valid for MessageConsumers/Producers. Both of them are bound to a session, so if a Session is lightweight to open then these objects will be as well.<p>
There is however a third solution; a custom DestinationResolver. The DestinationResolver is the abstraction that takes care of going from a String to a Destination. The default (<a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/support/destination/DynamicDestinationResolver.html">DynamicDestinationResolver</a>) uses the createConsumer(javax.jms.Destination) on the JMS Session to transform, but it does however not cache the resulting Destination. However, if your Destinations are configured on the application server as resources, you can (besides using Spring's JNDI support and injection the Destination directly) also use <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/support/destination/JndiDestinationResolver.html">JndiDestinationResolver</a>. This resolver will treat the supplied String as a JNDI location (instead of physical destination name) and perform a lookup for you. By default it will cache the resulting Destination, avoiding any subsequent JNDI lookups. Now, one can also configure JndiDestinationResolver as a caching decorator for the DynamicDestinationResolver. If you set fallback to true, it will first try to use the String as a location to lookup from JNDI, if that fails it will pass our String along to DynamicDestinationResolver using the JMS API to transform our String to a Destination. The resulting Destination is in both cases cached and thus a next request for the same Destination will be served from the cache. With this resolver there is a solution out of the box without having to write any code:
<pre class="brush: xml">
<bean id="cachingDestinationResolver" class="org.springframework.jms.support.destination.JndiDestinationResolver">
<property name="cache" value="true"/>
<property name="fallbackToDynamicDestination" value="true"/>
</bean>
<bean id="infra.abstractMessageListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="destinationResolver" ref="cachingDestinationResolver"/>
...
</bean>
</pre>
The JndiDestinationResolver is thread safe by internally using a ConcurrentHasmap to store the bindings. A JMS Destination is on itself thread safe according to the JMS 1.1 specification (2.8 Multithreading) and can safely be cached:<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-YRrWHJ6iqcM/UVltDJHv16I/AAAAAAAAAVA/YATJMVF2AuA/s1600/jmsObjects.png" imageanchor="1"><img border="0" src="http://4.bp.blogspot.com/-YRrWHJ6iqcM/UVltDJHv16I
/AAAAAAAAAVA/YATJMVF2AuA/s320/jmsObjects.png"/></a>
</div>
<p>
This is again a nice example on how simple things can sometimes have an important impact. This time the solution was straightforward thanks to Spring. It would however been a better idea to make the caching behaviour the default as this would decouple it from any provider specific quirks in looking up the destination. The reason this isn't the default is probably because the DefaultMessageListenerContainer supports <a target="_blank" href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/jms/listener/AbstractMessageListenerContainer.html#setDestinationName%28java.lang.String%29">changing the destination on the fly</a> (using JMX for example): <p>
<pre class="brush: plain">
Note: The destination may be replaced at runtime, with the listener container picking up the new destination immediately (works e.g. with DefaultMessageListenerContainer, as long as the cache level is less than CACHE_CONSUMER). However, this is considered advanced usage; use it with care!
</pre>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-66898815954605215552013-03-04T12:04:00.001+01:002013-03-15T08:39:16.273+01:00Bulk fetching with HibernateIf you need to process large database result sets from Java you can opt for JDBC to give you the low level control required. On the other hand if you are already using an ORM in your application falling back to JDBC might imply some extra pain. You would be losing features such as optimistic locking, caching, automatic fetching when navigating the domain model and so forth. Fortunately most ORMs, like Hibernate, have some options to help you with that. While these techniques are not new, there are a couple of possibilities to choose from.<p>
A simplified example; let's assume we have a table (mapped to class "DemoEntity") with 100.000 records. Each record consists of a single column (mapped to the property "property" in DemoEntity) holding some random alphanumerical data of about ~2KB. The JVM is ran with -Xmx250m. Let's assume that 250MB is the overall maximum memory that can be assigned to the JVM on our system. Your job is to read all records currently in the table, doing some not further specified processing, and finally store the result. We'll assume that the entities resulting from our bulk operation are not modified. To start we'll try the obvious first, performing a query to simply retrieve all data:
<pre class="brush: java;">
new TransactionTemplate(txManager).execute(new TransactionCallback<Void>() {
@Override
public Void doInTransaction(TransactionStatus status) {
Session session = sessionFactory.getCurrentSession();
List<DemoEntity> demoEntitities = (List<DemoEntity>) session.createQuery("from DemoEntity").list();
for(DemoEntity demoEntity : demoEntitities){
//Process and write result
}
return null;
}
});
</pre>
After a couple of seconds:
<p>
<pre>Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded</pre>
<p>
<a href="http://3.bp.blogspot.com/-pmT8fQhjADE/UTR8S6wVbTI/AAAAAAAAATs/F2k-SPqTcI0/s1600/Screenshot.png" imageanchor="1" ><img border="0" src="http://3.bp.blogspot.com/-pmT8fQhjADE/UTR8S6wVbTI/AAAAAAAAATs/F2k-SPqTcI0/s600/Screenshot.png" /></a><p>
Clearly this won't cut it. To fix this we will be switching to Hibernate scrollable result sets as probably most developers are aware of. The above example instructs hibernate to execute the query, map the entire results to entities and return them. When using scrollable result sets records are transformed to entities one at a time:
<pre class="brush: java; highlight: [5,8,12]">
new TransactionTemplate(txManager).execute(new TransactionCallback<Void>() {
@Override
public Void doInTransaction(TransactionStatus status) {
Session session = sessionFactory.getCurrentSession();
ScrollableResults scrollableResults = session.createQuery("from DemoEntity").scroll(ScrollMode.FORWARD_ONLY);
int count = 0;
while (scrollableResults.next()) {
if (++count > 0 && count % 100 == 0) {
System.out.println("Fetched " + count + " entities");
}
DemoEntity demoEntity = (DemoEntity) scrollableResults.get()[0];
//Process and write result
}
return null;
}
});
</pre>
After running this we get: <p>
<pre>
...
Fetched 49800 entities
Fetched 49900 entities
Fetched 50000 entities
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
</pre>
<a href="http://1.bp.blogspot.com/-1tSO33MsriM/UTR8ZjP-hXI/AAAAAAAAAT0/keoDlvHC02g/s1600/Screenshot-1.png" imageanchor="1" ><img border="0" src="http://1.bp.blogspot.com/-1tSO33MsriM/UTR8ZjP-hXI/AAAAAAAAAT0/keoDlvHC02g/s600/Screenshot-1.png" /></a><p>
Although we are using a scrollable result set, every returned object is an attached object and becomes part of the persistence context (aka session). The result is actually the same as our first example in which we used "<b>session.createQuery("from DemoEntity").list()</b>". However, with that approach we had no control; everything happens behind the scenes and you get a list back with all the data if hibernate has done its job. using a scrollable result set on the other hand gives us a hook into the retrieval process and allows us to free memory up when needed. As we have seen it does not free up memory automatically, you have to instruct Hibernate to actually do it. Following options exist:<p>
<ul>
<li>Evicting the object from the persistent context after processing it</li>
<li>Clearing the entire session every now and then</li>
</ul>
We will opt for the first. In the above example under line 13 (<i>//Process and write result</i>) we'll add:
<pre class="brush: java;">
session.evict(demoEntity);
</pre>
Important:
<ul>
<li>If you were to perform any modification to the entity (or entities it has associations with that are cascade evicted alongside), make sure to flush the session PRIOR evicting or clearing, otherwise queries hold back because of Hibernate's write behind will not be sent to the database</li>
<li>Evicting or clearing does not remove the entities from second level cache. If you enabled second level cache and are using it and you want to remove them as well use the desired <b>sessionFactory.getCache().evictXxx()</b> method</li>
<li>From the moment you evict an entity it will be no longer attached (no longer associated with a session). Any modification done to the entity at that stage will no longer be reflected to the database automatically. If you are using lazy loading, accessing any property that was not loaded prior the eviction will yield the famous org.hibernate.LazyInitializationException. So basically, make sure the processing for that entity is done (or it is at least initialized for further needs) before you evict or clear</li>
</ul>
After we run the application again, we see that it now successfully executes:
<pre>
...
Fetched 99800 entities
Fetched 99900 entities
Fetched 100000 entities
</pre>
<a href="http://4.bp.blogspot.com/-LQLJB9FsZ34/UTR8hHo-arI/AAAAAAAAAUA/DDTNxaTg_vA/s1600/Screenshot-2.png" imageanchor="1" ><img border="0" src="http://4.bp.blogspot.com/-LQLJB9FsZ34/UTR8hHo-arI/AAAAAAAAAUA/DDTNxaTg_vA/s600/Screenshot-2.png" /></a><p>
Btw; you can also set the query read-only allowing hibernate to perform some extra optimizations:
<pre class="brush: java;">
ScrollableResults scrollableResults = session.createQuery("from DemoEntity").setReadOnly(true).scroll(ScrollMode.FORWARD_ONLY);
</pre>
Doing this only gives a very marginal difference in memory usage, in this specific test setup it enabled us to read about 300 entities extra with the given amount of memory. Personally I would not use this feature merely for memory optimizations alone but only if it suits in your overall immutability strategy. With hibernate you have different options to make entities read-only: on the entity itself, the overall session read-only and so forth. Setting read only false on the query individually is probably the least preferred approach. (eg. entities loaded in the session before will remain unaffected, possibly modifiable. Lazy associations will be loaded modifiable even if the root objects returned by the query are read only).
<p>
Ok, we were able to process our 100.000 records, life is good. But as it turns out Hibernate has another another option for bulk operations: the stateless session. You can obtain a scrollable result set from a stateless session the same way as from a normal session. A stateless session lies directly above JDBC. Hibernate will run in nearly "all features disabled" mode. This means no persistent context, no 2nd level caching, no dirty detection, no lazy loading, basically no nothing. From the javadoc:
<pre>
/**
* A command-oriented API for performing bulk operations against a database.
* A stateless session does not implement a first-level cache nor interact with any
* second-level cache, nor does it implement transactional write-behind or automatic
* dirty checking, nor do operations cascade to associated instances. Collections are
* ignored by a stateless session. Operations performed via a stateless session bypass
* Hibernate's event model and interceptors. Stateless sessions are vulnerable to data
* aliasing effects, due to the lack of a first-level cache. For certain kinds of
* transactions, a stateless session may perform slightly faster than a stateful session.
*
* @author Gavin King
*/
</pre>
The only thing it does is transforming records to objects. This might be an appealing alternative because it helps you getting rid of that manual evicting/flushing:
<pre class="brush: java; highlight: [7,20]">
new TransactionTemplate(txManager).execute(new TransactionCallback<Void>() {
@Override
public Void doInTransaction(TransactionStatus status) {
sessionFactory.getCurrentSession().doWork(new Work() {
@Override
public void execute(Connection connection) throws SQLException {
StatelessSession statelessSession = sessionFactory.openStatelessSession(connection);
try {
ScrollableResults scrollableResults = statelessSession.createQuery("from DemoEntity").scroll(ScrollMode.FORWARD_ONLY);
int count = 0;
while (scrollableResults.next()) {
if (++count > 0 && count % 100 == 0) {
System.out.println("Fetched " + count + " entities");
}
DemoEntity demoEntity = (DemoEntity) scrollableResults.get()[0];
//Process and write result
}
} finally {
statelessSession.close();
}
}
});
return null;
}
});
</pre>
<a href="http://4.bp.blogspot.com/-vOiakyV7w4k/UTR8nfaskHI/AAAAAAAAAUI/GYxtBecbzMI/s1600/Screenshot-3.png" imageanchor="1" ><img border="0" src="http://4.bp.blogspot.com/-vOiakyV7w4k/UTR8nfaskHI/AAAAAAAAAUI/GYxtBecbzMI/s600/Screenshot-3.png" /></a><p>
Besides the fact that the stateless session has the most optimal memory usage, using the it has some side effects. You might have noticed that we are opening a stateless session and closing it explicitly: there is no <i>sessionFactory.getCurrentStatelessSession()</i> nor (at the time of writing) any Spring integration for managing the stateless session.Opening a stateless session allocates a new java.sql.Connection by default (if you use <i>openStatelessSession()</i>) to perform its work and therefore indirectly spawns a second transaction. You can mitigate these side effects by using the Hibernate work API as in the example which supplies the current Connection and pass it along to openStatelessSession(Connection connection). Closing the session in the finally has no impact on the physical connection since that is captured by the Spring infrastructure: only the logical connection handle is closed and a new logical connection handle was created when opening the stateless session.
<p>
Also note that you have to deal with closing the stateless session yourself and that the above example is only good for read-only operations. From the moment you are going to modify using the stateless session there are some more caveats. As said before, hibernate runs in "all feature disabled" mode and as a direct consequence entities are returned in detached state.
For each entity you modify, you'll have to call: <i>statelessSession.update(entity)</i> explicitly. First I tried this for modifying an entity:
<pre class="brush: java; highlight: [11]">
new TransactionTemplate(txManager).execute(new TransactionCallback<Void>() {
@Override
public Void doInTransaction(TransactionStatus status) {
sessionFactory.getCurrentSession().doWork(new Work() {
@Override
public void execute(Connection connection) throws SQLException {
StatelessSession statelessSession = sessionFactory.openStatelessSession(connection);
try {
DemoEntity demoEntity = (DemoEntity) statelessSession.createQuery("from DemoEntity where id = 1").uniqueResult();
demoEntity.setProperty("test");
statelessSession.update(demoEntity);
} finally {
statelessSession.close();
}
}
});
return null;
}
});
</pre>
The idea is that we open a stateless session with the existing database Connection. As the StatelessSession javadoc indicates that no write behind occurs, I was convinced that each statement performed by the stateless session would be sent directly to the database. Eventually when the transaction (started by the TransactionTemplate) would be committed the results would become visible in the database. However, hibernate does BATCH statements using a stateless session. I'm not 100% sure what the difference is between batching and write behind, but the result is the same and thus contra dictionary with the javadoc as statements are queued and flushed at a later time. So, if you don't do anything special, statements that are batched will not be flushed and this is what happened in my case: the "statelessSession.update(demoEntity);" was batched and never flushed. One way to force the flush is to use the hibernate transaction API:
<pre class="brush: java;">
StatelessSession statelessSession = sessionFactory.openStatelessSession();
statelessSession.beginTransaction();
...
statelessSession.getTransaction().commit();
...
</pre>
While this works, you probably don't want to start controlling your transactions programatically just because you are using a stateless session. Also, doing this we are again running our stateless session work in a second transaction scenario since we didn't pass along our Connection and thus a new database connection will be acquired. The reason we can't pass along the outer Connection is because if we commit the inner transaction (the "stateless session transaction") and it would be using the same connection as the outer transaction (started by the TransactionTemplate) it would break the outer transaction atomicity as statements from the outer transaction sent to database would be committed along with the inner transaction. So not passing along the connections means opening a new connection and thus creating a second transaction. A better alternative would be just to trigger Hibernate to flush the stateless session. However, statelessSession has no "flush" method to manually trigger a flush. A solution here is to depend a bit on the Hibernate internal API. This solution makes the manual transaction handling and the second transaction obsolete: all statements become part of our (one and only) outer transaction:
<pre class="brush: java; highlight: [6]">
StatelessSession statelessSession = sessionFactory.openStatelessSession(connection);
try {
DemoEntity demoEntity = (DemoEntity) statelessSession.createQuery("from DemoEntity where id = 1").uniqueResult();
demoEntity.setProperty("test");
statelessSession.update(demoEntity);
((TransactionContext) statelessSession).managedFlush();
} finally {
statelessSession.close();
}
</pre>
Fortunately there is an even better solution very recently posted on the Spring jira: <a target="_blank" href="https://jira.springsource.org/browse/SPR-2495">https://jira.springsource.org/browse/SPR-2495</a>
This is not yet part of Spring, but the factory bean implementation is pretty straight forward: <a target="_blank" href="https://jira.springsource.org/secure/attachment/20890/StatelessSessionFactoryBean.java">StatelessSessionFactoryBean.java</a> when using this you could simple inject the StatelessSession:
<pre class="brush: java;">
@Autowired
private StatelessSession statelessSession;
</pre>
It will inject a stateless session proxy which is equivalent to the way the normal "current" session works (with the minor difference that you inject a SessionFactory and need to obtain the currentSession each time). When the proxy is invoked it will lookup the stateless session bound to the running transaction. If none exists already it will create one with the same connection as the normal session (like we did in the example) and register a custom transaction synchronization for the stateless session. When the transaction is committed the stateless session is flushed thanks to the synchronization and finally closed. Using this you can inject the stateless session directly and use it as a current session (or the same way as you would inject a JPA PeristentContext for that matter). This relieves you from dealing with the opening and closing of the stateless session and having to deal with one way or the other to make it flush. The implementation is JPA aimed, but the JPA part is limited to obtaining the physical connection in obtainPhysicalConnection(). You can easily leave out the EntityManagerFactory and get the physical connection directly from the Hibernate session.
<p>
Very careful conclusion: it is clear that the best approach will depend on your situation. If you use the normal session you will have to deal with eviction yourself when reading or persisting entities. Besides the fact you have to do this manually, it might also impact further use of the session if you have a mixed transaction; you both perform 'bulk' and 'normal' operations in the same transaction. If you continue with the normal operations you will have detached entities in your session which might lead to unexpected results (as dirty detection will no longer work and so forth). On the other hand you will still have the major hibernate benefits (as long as the entity isn't evicted) such as lazy loading, caching, dirty detection and the likes. Using the stateless session at the time of writing requires some extra attention on managing it (opening, closing and flushing) which can also be error prone. In the assumption you can proceed with the proposed factory bean, you have a very bare bone session which is separately from your normal session but still participating in the same transaction. With this you have a powerful tool to perform bulk operations without having to think about memory management. The downside is that you don't have any other hibernate functionality available.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com18tag:blogger.com,1999:blog-3406746278542965235.post-1619710913908554492013-02-18T18:23:00.002+01:002013-02-18T18:30:14.770+01:00Unit (resource/frontend) testing templateUnit testing requires a lot of time and investment. One of the important things to safe keep this investment is building on good infrastructure.
While we are all used to design production code, designing our tests and test infrastructure is in many cases forgotten or the time needed is simply not there.
In many cases the requirements for the test set also grow (just like real requirements). Sometimes testing can even become a project inside the project.
I have gathered some testing practices I've became used with over the years and put them in a small project, which you can find here:
<a target="_blank" href="https://github.com/koen-serneels/unit-testing">github unit-testing</a><p>
It is a Spring/Spring MVC webapp using JPA (Hibernate as provider) for persistence and TestNG, Spring test support, selenium2 for testing.
There are 3 types of test covered: basic unit test, unit resource test and front end test (unit resource tests with a flavor).
<p>
The basic unit tests give an idea how you can use Spring within your tests or use its MVC test extensions to test for example application controllers .
The unit resources tests give a strategy for testing code that is depended on a resource, but cannot be fully tested without that resource (eg. dummies, mocks or stubs are not the best alternative).
For example repositories depending on a RDBMS. In that case we work with an embedded database which allows to test our repositories very close to their production environment.
<p>
The front end tests require some extra features as they need to communicate with the database booted inside the deployed application.
Also, to make it interesting I have put a repository that depends on an "not further specified" sub system for which no in memory equivalents exists.
A stub is put into place in the deployed application which is remotely controllable from within the test.
This allows you to have the data setup from the test for both resources (the one requiring a database and this one requiring the sub system) *in* the test.
The data for both resources is setup and removed in scope of each test method (or tat least each test).
Alternatives likes simulators have the data deployed together with the application which makes it brittle for changes.
(eg. the data is automatically shared for all tests and packed together in one place)
<p>
The project is Maven enabled; importing it should be zero hassle. Surefire and failsafe (+cargo) are also configured, so all your tests (including the front end tests) are executed when performing a build with Maven. Since there are no dependencies whatsoever (besides the one defined in the pom) you can add this kind of project directly to your CI hassle-free.<p>
There is also a PDF explaining some of the ideas, but it was put together in short time, so don't expect too much from it. Feel free to share your testing strategies!Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com1tag:blogger.com,1999:blog-3406746278542965235.post-7246652788711472302013-02-02T20:08:00.000+01:002013-02-03T10:59:46.561+01:00Thermal shutdownWhile running a CPU intensive process last week my laptop decided to instantly power off on me. Since the battery was not installed I thought it was probably a power glitch, so I turned it back on and restarted the process. After a few minutes it powered off again, so far for the power glitch.<p>
Thinking it might be temperature related I installed sensors and Computer Temperature Monitor for Ubuntu. As it turned out my CPU temperature was pretty high. When the process started it was around 90°C, slowly but steadily progressing towards 100°C. After that it was a matter of seconds before it powered off. Google confirmed my suspicion that 100°C is indeed very high for a laptop core. Running the same process on another comparable laptop ran a whopping 20°C cooler. It was now for sure that it was a thermal shutdown.<p>
Eliminating the easiest thing first seemed a sane thing to do; for me this would be any influence imposed by software. First I booted Windows running the same process there again and gathering temperature readings using Sisoft Sandra. The temperature matched with those gathered in Ubuntu, powering of the laptop again after some minutes. My final attempt was to update the BIOS but again without any noticeable differences.<p>
On a side note; the laptop was docked on a docking station when performing these tests.
The docking station allows even more distance between the air-intake (located on the bottom side of the laptop) and the surface. Room temperature was normal and the (one and only) fan was also running and speeding up normally (as far as my audiovisual inspection powers could tell). Giving these standard conditions, and the fact that I was not even using everything (the GPU for example) a thermal shutdown was certainly not to be expected. Next I started out with the hardware. Again the easiest thing first: clean the air vents. Got myself a can with compressed air to clean the vents:<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-_tV1mcSNhDI/UQ1dFdY7UuI/AAAAAAAAAR8/rwgNm7ZZWOs/s1600/IMG_4442.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="240" src="http://4.bp.blogspot.com/-_tV1mcSNhDI/UQ1dFdY7UuI/AAAAAAAAAR8/rwgNm7ZZWOs/s320/IMG_4442.JPG" /></a></div>
<p>
There was some dust, but not much. As expected this did not solve the heating problem. I repeated the same procedure, but this time with the keyboard and switch/led cover removed for easier access:
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-QQ0Bso2lPcs/UQ1svh60SWI/AAAAAAAAATc/gaqqcJCCxi0/s1600/IMG_4447.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="240" width="320" src="http://2.bp.blogspot.com/-QQ0Bso2lPcs/UQ1svh60SWI/AAAAAAAAATc/gaqqcJCCxi0/s320/IMG_4447.JPG" /></a></div>
Next I emptied some air capsules used for inflating the tires of my MTB on the fan and vents. They produce a pretty high amount of air displacement:-) <p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-ZzOWSWJrJuY/UQ1gL5EPX1I/AAAAAAAAASM/CtOpQ8n4jjg/s1600/aircan.jpg" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="163" width="320" src="http://2.bp.blogspot.com/-ZzOWSWJrJuY/UQ1gL5EPX1I/AAAAAAAAASM/CtOpQ8n4jjg/s320/aircan.jpg" /></a></div>
<p>
After everything was defrosted again the vents were now clean for sure. But no improvements on the horizon for this skipper!<p>
My next idea was to decided if the fan was behaving properly. Easier said then done, since I was unable to get any RPM readings. ACPI, closed BIOSes and the likes are apparently withholding me from getting that information. My 15 year old self build desktop had RPM readings of the CPU fan, GPU fan and every case fan. But as it seems, we no longer need health indication of our fans these days. The closest thing I was able to do was comparing the air displacement of the fan with the fan of my comparable laptop.<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-E4xhRpAXTqk/UQ1hA3BepdI/AAAAAAAAASY/zroeGBLSav8/s1600/IMG_4455.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="240" src="http://1.bp.blogspot.com/-E4xhRpAXTqk/UQ1hA3BepdI/AAAAAAAAASY/zroeGBLSav8/s320/IMG_4455.JPG" /></a></div>
<p>
As it turned out, on top speed the air displacement was similar between the two laptops. This gave me a satisfactoriness indication that the fan/fan-controller were OK. All other inspections turned out to be OK as well; heat sinks were still fixed and heating up properly, so I was running out of ideas.<p>
Getting annoyed I googled some more and stumbled across a blog about installing a custom heat sink for my model and the steps it involved. One of those steps is as most of you know applying thermal paste between the core and the heat sink. Thermal paste helps in pushing out remaining air between core and heat sink for optimal heat transfer. As a fan issue was now excluded, it could well be that the heat was perhaps not even properly being transfered to be vented out... I also realized that the temperature fluctuations of the CPU were indeed a bit strange. Idle it was around 40°C, but when the process started the temperature went up so fast that the fan lagged behind some seconds before shifting to top speed. When the fan finally reached its top speed the temperature increase was slowed down, but in the mean time the CPU had already ~90°C.
<p>
I decided to renew the thermal paste as a final act of desperation. Again easier said then done, at least for me, since the heat sink screws are torx, dammit! So, first to the home depot buying some torx equipment, and then to the computer shop getting some silver based thermal paste. <p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-yQRh8NSg7MI/UQ1hnzpvTlI/AAAAAAAAASo/ZSJqu2MLoQs/s1600/IMG_4446.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="240" src="http://2.bp.blogspot.com/-yQRh8NSg7MI/UQ1hnzpvTlI/AAAAAAAAASo/ZSJqu2MLoQs/s320/IMG_4446.JPG" /></a></div>
<p>
When I removed the heat sink I saw that they used ceramic based paste and that it was hardened out. A good indication I could be on to something since its not supposed to be hardened out, and certainly not on a only 1.5 year old laptop. Cleaned the paste of first:
<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-NA33XQuqFb4/UQ1iJbJwxLI/AAAAAAAAASw/C_p6cCSqJDA/s1600/IMG_4448.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="240" width="320" src="http://1.bp.blogspot.com/-NA33XQuqFb4/UQ1iJbJwxLI/AAAAAAAAASw/C_p6cCSqJDA/s320/IMG_4448.JPG" /></a></div>
<p>
Added new paste by putting a tiny drop in the middle on each IC surface. This is enough as when you install the heatsink it will (hopefully) spread evenly over its surface. To verify this I installed the heat sink, tightened the screws and then removed it again. In my case the surface is covered without the paste going over it, so that is what we want:
<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-JiyrNZD-ogc/UQ1i3NnL5uI/AAAAAAAAAS8/pZ_4EajBQUc/s1600/IMG_4450.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="240" width="320" src="http://3.bp.blogspot.com/-JiyrNZD-ogc/UQ1i3NnL5uI/AAAAAAAAAS8/pZ_4EajBQUc/s320/IMG_4450.JPG" /></a></div>
<p>
As it turned out I was lucky this time, after applying the new thermal paste the CPU temperature was now steady at around 80°C when on full load. Nice.
<p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-KN2mCjKaQsI/UQ1i9nwDJeI/AAAAAAAAATI/92RUQvDsw1s/s1600/Screenshot.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="326" width="400" src="http://1.bp.blogspot.com/-KN2mCjKaQsI/UQ1i9nwDJeI/AAAAAAAAATI/92RUQvDsw1s/s400/Screenshot.png" /></a></div>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com2tag:blogger.com,1999:blog-3406746278542965235.post-33543190548006451262012-11-19T17:40:00.003+01:002012-11-19T20:37:24.965+01:00JavaFX 2 with SpringI'm going to start this one with a bold statement: I always liked Java Swing, or applets for that matter. There, I said it.
If I perform some self analysis, this admiration probably started when I got introduced to Java.
Swing was (practically) the first thing I ever did with Java that gave some statisfactionary result and made me able to do something with the language at the time.
When I was younger we build home-brew fat clients to manage our 3.5" floppy/CD collection (written in VB and before that in basic) this probably also played a role.
<br/><br/>
Anyway, enough about my personal rareness. Fact is that Swing has helped many build great applications but as we all know Swing has it drawbacks.
For starters it hasn't been evolving since well, a long time. It also requires a lot of boiler plate code if you want to create high quality code.
It comes shipped with some quirky design "flaws", lacks out of the box patterns such as MVC.
Styling is a bit of a limitation since you have to fall back on the limited L&F architecture, I18N is not build in by default and so on.
One could say that developing Swing these days is well, basically going back into time.
<br/><br/>
Fortunately Oracle tried to change this some years ago by Launching JavaFX.
I recall getting introduced to JavaFX on Devoxx (or Javapolis as it was named back then).
The nifty demo's looked very promising, so I was glad to see that a Swing successor was finally on its way.
This changed from the moment I saw its internals. One of its major drawbacks was that it was based on a dark new syntax (called JavaFX script).
In case you have never seen JavaFX script; it looks like a bizarre breed between Java, JSON and JavaScript.
Although it is compiled to Java byte-code, and you could use the Java API's from it, integration with Java was never really good.
<br/><br/>
The language itself (although pretty powerful) required you to spend a lot of time understanding the details, for ending up with, well, again source code, but this time less manageable and supported then plain Java code.
As it turned out, I wasn't the only one. A lot of people felt the same (for sure there were other reasons as well) and JavaFX never was a great success.
<br/><br/>
However, a while ago Oracle changed the tide by introducing JavaFX 2.
<br>
First of all they got rid of JavaFX script (which is no longer supported) and turned it into a real native Java SE API (JavaFX 2.2.3 is part of the Java 7 SE update 6)
. The JavaFX API now looks more like the familiar Swing API, which is a good thing.
It gives you layout managers lookalikes, event listeners, and all those other components you were so used to, but even better.
So if you want you can code JavaFX like you did Swing you can, albeit with slightly different syntax and improved architecture.
It is also possible now to <a target="_blank" href="http://docs.oracle.com/javafx/2/swing/overview.htm#CJAHBAHA">intermix existing Java Swing applications with JavaFX</a>.<br/><br/>
But there is more. They introduced an XML based markup language that allows you to describe the view. This has some advantages, first of all coding in XML works faster then Java.
XML can be more easily be generated then Java and the syntax for describing a view is simply more compact.
It is also more intuitive to express a view using some kind of markup, especially if you ever did some web development before.
So, one can have the view described in FXML (thats how its called), the application controllers separate from the view, both in Java, and your styling in CSS (yeah, so no more L&F, CSS support is standard).
You can still embed Java (or other languages) directly in the FXML; but this is probably not what you want (scriptlet anti-pattern).
Another nice thing is support for binding. You can bind each component in your view to the application controller by putting an fx:id attribute on the view component and an @FXML annotation on the instance variable in the application controller.
The corresponding element will then be auto injected, so you can change its data or behavior from inside your application controller.
It also turns out that with some lines of code you can painlessly integrate the DI framework of your choice, isn't that sweet?
<br/><br/>
And what about the tooling? <br/>
Well, first of all there is a plug-in for Eclipse (fxclipse) which will render you FXML on the fly. You can install it via Eclipse market place:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-E7kb_CDIY8U/UKpbtCztK5I/AAAAAAAAAQ0/9xGeHW-THNo/s1600/market_place.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="195" width="400" src="http://2.bp.blogspot.com/-E7kb_CDIY8U/UKpbtCztK5I/AAAAAAAAAQ0/9xGeHW-THNo/s400/market_place.png" /></a></div>
<br/>
The plug-in will render any adjustment you make immediately:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-h1qcElPqv8Q/UKpaN-n0GBI/AAAAAAAAAPU/XUe3F5NDag4/s1600/fxclipse.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="132" width="400" src="http://4.bp.blogspot.com/-h1qcElPqv8Q/UKpaN-n0GBI/AAAAAAAAAPU/XUe3F5NDag4/s400/fxclipse.png" /></a></div><br/><br/>
Note that you need at least JDK7u6 for this plug-in to work. If your JDK is too old you'll get an empty pane in eclipse. Also, if you create a JavaFX project I needed to put the jfxrt.jar manually on my build classpath. You'll find this file in %JAVA_HOME%/jre/lib.
<br/><br/>
Up until know the plug-in doesn't help you visually (by drag& drop) but that there a separate IDE: <a href="http://www.oracle.com/technetwork/java/javafx/tools/index.html" target="_blank">scene builder</a>. This builder is also integrated in Netbeans, for AFAIK there is no support for eclipse yet so you'll have to run it separately if you want to use it. The builder lets you develop FXML the visual way, using drag&drop. Nice detail; scene builder is in fact written in JavaFX. Then you also have a separate application called <a target="_blank" href="http://fxexperience.com/scenic-view/">scenic view</a> which does introspection on a running JavaFX application and shows how it is build up. You get a graph with the different nodes and their hierarchical structure. For each node you can see its properties and so forth:
<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-dH7A2yCsqPE/UKpe3XNjNdI/AAAAAAAAARI/1_qwJrKZs9k/s1600/scenic_view.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="250" width="400" src="http://1.bp.blogspot.com/-dH7A2yCsqPE/UKpe3XNjNdI/AAAAAAAAARI/1_qwJrKZs9k/s400/scenic_view.png" /></a></div>
<br/>
Ok, so lets start with some code examples. The first thing I did was design my demo application in scene builder:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-qdCbZTncErA/UKpah_zjsNI/AAAAAAAAAPg/ePMx9oMyn6I/s1600/scene_builder_1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="283" width="400" src="http://3.bp.blogspot.com/-qdCbZTncErA/UKpah_zjsNI/AAAAAAAAAPg/ePMx9oMyn6I/s400/scene_builder_1.png" /></a></div><br/>
I did this graphically by d&d the containers/controlers on to the view. I also gave the controls that I want to bind to my view and fx:id, you can do that also via scene builder:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-m1o5lbkI8Ic/UKpamrCmN4I/AAAAAAAAAPs/ns-bAGPtx6k/s1600/scene_builder_2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="263" width="400" src="http://1.bp.blogspot.com/-m1o5lbkI8Ic/UKpamrCmN4I/AAAAAAAAAPs/ns-bAGPtx6k/s400/scene_builder_2.png" /></a></div><br/>
For the buttons in particular I also added an onAction (which is the method that should be executed on the controller once the button is clicked):<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-XhaiQnEsCLM/UKpaq-f9QpI/AAAAAAAAAP4/gImmT8I52b8/s1600/scene_builder_3.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="263" width="400" src="http://1.bp.blogspot.com/-XhaiQnEsCLM/UKpaq-f9QpI/AAAAAAAAAP4/gImmT8I52b8/s400/scene_builder_3.png" /></a></div><br/>
Next I added the controller manually in the source view in eclipse. There can only be one controller per FXML and it should be declared in the top level element. I made two FXML's, one that represents the main screen and one that acts as the menu bar. You probably want a division of your logic in multiple controllers, rather then stuffing to much in a single controller – single responsibility is a good design guideline here. The first FXML is “search.fxml” and represents the search criteria and result view:
<pre class="brush: xml; highlight: [11, 17]">
<?xml version="1.0" encoding="UTF-8"?>
<?import java.lang.*?>
<?import java.util.*?>
<?import javafx.scene.control.*?>
<?import javafx.scene.control.Label?>
<?import javafx.scene.control.cell.*?>
<?import javafx.scene.layout.*?>
<?import javafx.scene.paint.*?>
<StackPane id="StackPane" maxHeight="-Infinity" maxWidth="-Infinity" minHeight="-Infinity" minWidth="-Infinity" prefHeight="400.0" prefWidth="600.0" xmlns:fx="http://javafx.com/fxml" fx:controller="be.error.javafx.controller.SearchController">
<children>
<SplitPane dividerPositions="0.39195979899497485" focusTraversable="true" orientation="VERTICAL" prefHeight="200.0" prefWidth="160.0">
<items>
<GridPane fx:id="grid" prefHeight="91.0" prefWidth="598.0">
<children>
<fx:include source="/menu.fxml"/>
<GridPane prefHeight="47.0" prefWidth="486.0" GridPane.columnIndex="1" GridPane.rowIndex="5">
<children>
<Button fx:id="clear" cancelButton="true" mnemonicParsing="false" onAction="#clear" text="Clear" GridPane.columnIndex="1" GridPane.rowIndex="1" />
<Button fx:id="search" defaultButton="true" mnemonicParsing="false" onAction="#search" text="Search" GridPane.columnIndex="2" GridPane.rowIndex="1" />
</children>
<columnConstraints>
<ColumnConstraints hgrow="SOMETIMES" maxWidth="338.0" minWidth="10.0" prefWidth="338.0" />
<ColumnConstraints hgrow="SOMETIMES" maxWidth="175.0" minWidth="0.0" prefWidth="67.0" />
<ColumnConstraints hgrow="SOMETIMES" maxWidth="175.0" minWidth="10.0" prefWidth="81.0" />
</columnConstraints>
<rowConstraints>
<RowConstraints maxHeight="110.0" minHeight="10.0" prefHeight="10.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="72.0" minHeight="10.0" prefHeight="40.0" vgrow="SOMETIMES" />
</rowConstraints>
</GridPane>
<Label alignment="CENTER_RIGHT" prefHeight="21.0" prefWidth="101.0" text="Product name:" GridPane.columnIndex="0" GridPane.rowIndex="1" />
<TextField fx:id="productName" prefWidth="200.0" GridPane.columnIndex="1" GridPane.rowIndex="1" />
<Label alignment="CENTER_RIGHT" prefWidth="101.0" text="Min price:" GridPane.columnIndex="0" GridPane.rowIndex="2" />
<Label alignment="CENTER_RIGHT" prefWidth="101.0" text="Max price:" GridPane.columnIndex="0" GridPane.rowIndex="3" />
<TextField fx:id="minPrice" prefWidth="200.0" GridPane.columnIndex="1" GridPane.rowIndex="2" />
<TextField fx:id="maxPrice" prefWidth="200.0" GridPane.columnIndex="1" GridPane.rowIndex="3" />
</children>
<columnConstraints>
<ColumnConstraints hgrow="SOMETIMES" maxWidth="246.0" minWidth="10.0" prefWidth="116.0" />
<ColumnConstraints fillWidth="false" hgrow="SOMETIMES" maxWidth="537.0" minWidth="10.0" prefWidth="482.0" />
</columnConstraints>
<rowConstraints>
<RowConstraints maxHeight="64.0" minHeight="10.0" prefHeight="44.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="68.0" minHeight="0.0" prefHeight="22.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="68.0" minHeight="10.0" prefHeight="22.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="68.0" minHeight="10.0" prefHeight="22.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="167.0" minHeight="10.0" prefHeight="14.0" vgrow="SOMETIMES" />
<RowConstraints maxHeight="167.0" minHeight="10.0" prefHeight="38.0" vgrow="SOMETIMES" />
</rowConstraints>
</GridPane>
<StackPane prefHeight="196.0" prefWidth="598.0">
<children>
<TableView fx:id="table" prefHeight="200.0" prefWidth="200.0">
<columns>
<TableColumn prefWidth="120.0" resizable="true" text="OrderId">
<cellValueFactory>
<PropertyValueFactory property="orderId" />
</cellValueFactory>
</TableColumn>
<TableColumn prefWidth="120.0" text="CustomerId">
<cellValueFactory>
<PropertyValueFactory property="customerId" />
</cellValueFactory>
</TableColumn>
<TableColumn prefWidth="120.0" text="#products">
<cellValueFactory>
<PropertyValueFactory property="productsCount" />
</cellValueFactory>
</TableColumn>
<TableColumn prefWidth="120.0" text="Delivered">
<cellValueFactory>
<PropertyValueFactory property="delivered" />
</cellValueFactory>
</TableColumn>
<TableColumn prefWidth="120.0" text="Delivery days">
<cellValueFactory>
<PropertyValueFactory property="deliveryDays" />
</cellValueFactory>
</TableColumn>
<TableColumn prefWidth="150.0" text="Total order price">
<cellValueFactory>
<PropertyValueFactory property="totalOrderPrice" />
</cellValueFactory>
</TableColumn>
</columns>
</TableView>
</children>
</StackPane>
</items>
</SplitPane>
</children>
</StackPane>
</pre>
On line 11 you can see that I configured the application controller class that should be used with the view. On line 17 you can see the import of the separate menu.fxml which is shown here:
<pre class="brush: xml; highlight: [7]">
<?xml version="1.0" encoding="UTF-8"?>
<?import javafx.scene.control.*?>
<?import javafx.scene.layout.*?>
<?import javafx.scene.control.MenuItem?>
<Pane prefHeight="465.0" prefWidth="660.0" xmlns:fx="http://javafx.com/fxml" fx:controller="be.error.javafx.controller.FileMenuController">
<children>
<MenuBar layoutX="0.0" layoutY="0.0">
<menus>
<Menu mnemonicParsing="false" text="File">
<items>
<MenuItem text="Exit" onAction="#exit" />
</items>
</Menu>
</menus>
</MenuBar>
</children>
</Pane>
</pre>
On line 7 you can see that it uses a different controller.
In Eclipse, if you open the fxclipse view from the plug-in you will get the same rendered view as in scene builder. Its however convenient if you want to make small changes in the code to see them directly reflected:
The code for launching the application is pretty standard:
<pre class="brush: java;">
package be.error.javafx;
import javafx.application.Application;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.stage.Stage;
public class TestApplication extends Application {
private static final SpringFxmlLoader loader = new SpringFxmlLoader();
@Override
public void start(Stage primaryStage) {
Parent root = (Parent) loader.load("/search.fxml");
Scene scene = new Scene(root, 768, 480);
primaryStage.setScene(scene);
primaryStage.setTitle("JavaFX demo");
primaryStage.show();
}
public static void main(String[] args) {
launch(args);
}
}
</pre>
The only special thing to note is that we extend from Application. This is a bit boiler plate code which will for example make sure that creating of the UI happens on the JavaFX application thread. You might remember such stories from Swing, where every UI interaction needs to occur on the event dispatcher thread (EDT), this is the same with JavaFX. You are by default on the “right thread” when you are called back by the application (in for example action listeners alike methods). But if you start the application or perform long running tasks in separate threads you need to make sure you start UI interaction on the right thread. For swing you would use <a target="_blank" href="http://docs.oracle.com/javase/7/docs/api/javax/swing/SwingUtilities.html#invokeLater%28java.lang.Runnable%29">SwingUtilities.invokeLater()</a> for JavaFX: <a target="_blank" href="http://docs.oracle.com/javafx/2/api/javafx/application/Platform.html">Platform.runLater()</a>.<br/>
More special is our SpringFxmlLoader:
<pre class="brush: java;highlight: [22,23,24,25,26,27]">
package be.error.javafx;
import java.io.IOException;
import java.io.InputStream;
import javafx.fxml.FXMLLoader;
import javafx.util.Callback;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
public class SpringFxmlLoader {
private static final ApplicationContext applicationContext = new AnnotationConfigApplicationContext(SpringApplicationConfig.class);
public Object load(String url) {
try (InputStream fxmlStream = SpringFxmlLoader.class
.getResourceAsStream(url)) {
System.err.println(SpringFxmlLoader.class
.getResourceAsStream(url));
FXMLLoader loader = new FXMLLoader();
loader.setControllerFactory(new Callback<Class<?>, Object>() {
@Override
public Object call(Class<?> clazz) {
return applicationContext.getBean(clazz);
}
});
return loader.load(fxmlStream);
} catch (IOException ioException) {
throw new RuntimeException(ioException);
}
}
}
</pre>
The highlighted lines show the custom ControllerFactory. Without setting this JavaFX will simply instantiate the class you specified as controller in the FXML without anything special. In that case the class will not be Spring managed (unless you would be using CTW/LTW AOP). By specifying a custom factory we can define how the controller should be instantiated. In this case we lookup the bean from the application context.
Finally we have our two controllers, the SearchController:
<pre class="brush: java; highlight: [24, 26, 38, 42, 54]">
package be.error.javafx.controller;
import java.math.BigDecimal;
import java.net.URL;
import java.util.ResourceBundle;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.fxml.FXML;
import javafx.fxml.Initializable;
import javafx.scene.control.Button;
import javafx.scene.control.TableView;
import javafx.scene.control.TextField;
import org.apache.commons.lang.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import be.error.javafx.model.Order;
import be.error.javafx.model.OrderSearchCriteria;
import be.error.javafx.model.OrderService;
public class SearchController implements Initializable {
@Autowired
private OrderService orderService;
@FXML
private Button search;
@FXML
private TableView<Order> table;
@FXML
private TextField productName;
@FXML
private TextField minPrice;
@FXML
private TextField maxPrice;
@Override
public void initialize(URL location, ResourceBundle resources) {
table.setColumnResizePolicy(TableView.CONSTRAINED_RESIZE_POLICY);
}
public void search() {
OrderSearchCriteria orderSearchCriteria = new OrderSearchCriteria();
orderSearchCriteria.setProductName(productName.getText());
orderSearchCriteria
.setMaxPrice(StringUtils.isEmpty(minPrice.getText()) ? null:new BigDecimal(minPrice.getText()));
orderSearchCriteria
.setMinPrice(StringUtils.isEmpty(minPrice.getText()) ? null: new BigDecimal(minPrice.getText()));
ObservableList<Order> rows = FXCollections.observableArrayList();
rows.addAll(orderService.findOrders(orderSearchCriteria));
table.setItems(rows);
}
public void clear() {
table.setItems(null);
productName.setText("");
minPrice.setText("");
maxPrice.setText("");
}
}
</pre>
The highlighted lines in respective order:
<ul>
<li>Auto injection by Spring, this is our Spring managed service which we will use to lookup data from</li>
<li>Auto injection by JavaFX, our controls that we need to manipulate or read from in our controller</li>
<li>Special init method to initialize our table so columns will auto resize when the view is enlarged
<li>action listener style callback which is invoked when the search button is pressed</li>
<li>action listener style callback which is invoked when the clear button is pressed</li>
</ul>
Finally the FileMenuController which does nothing special besides closing our app:
<pre class="brush: java;">
package be.error.javafx.controller;
import javafx.application.Platform;
import javafx.event.ActionEvent;
public class FileMenuController {
public void exit(ActionEvent actionEvent) {
Platform.exit();
}
}
</pre>
And finally the (not so exciting) result:
<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-o5R2NUbs-Js/UKpa12--4mI/AAAAAAAAAQE/xUGr0xPNAIo/s1600/javafx_1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="264" width="400" src="http://2.bp.blogspot.com/-o5R2NUbs-Js/UKpa12--4mI/AAAAAAAAAQE/xUGr0xPNAIo/s400/javafx_1.png" /></a></div>
After searching:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-FOjO_vWS9ak/UKpa6R2LWVI/AAAAAAAAAQQ/kjQ-pPMu9xU/s1600/javafx_2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="264" width="400" src="http://1.bp.blogspot.com/-FOjO_vWS9ak/UKpa6R2LWVI/AAAAAAAAAQQ/kjQ-pPMu9xU/s400/javafx_2.png" /></a></div><br/>
Making view wider, also stretches the columns:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-Vq0LyC6HSIw/UKpbDtWisKI/AAAAAAAAAQc/6YyiMLiQXn8/s1600/javafx_3.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="182" width="400" src="http://1.bp.blogspot.com/-Vq0LyC6HSIw/UKpbDtWisKI/AAAAAAAAAQc/6YyiMLiQXn8/s400/javafx_3.png" /></a></div><br/>
The file menu allowing us the exit:<br/><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-EptVDc_WyA4/UKpbH6b04FI/AAAAAAAAAQo/dHW7bBQm738/s1600/javafx_4.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="163" width="121" src="http://4.bp.blogspot.com/-EptVDc_WyA4/UKpbH6b04FI/AAAAAAAAAQo/dHW7bBQm738/s400/javafx_4.png" /></a></div><br/>
After playing a bit with JavaFX2 I was pretty impressed. There are also more and more controls coming (I believe there is already a browser control and such). So I think we are on the right track here.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com8tag:blogger.com,1999:blog-3406746278542965235.post-20771950463694493322012-09-12T16:46:00.001+02:002012-09-12T21:05:45.753+02:00Hibernate's Map behaviorHibernate has a map construct which makes mapping key/value pairs rather elegant.
Let's take a "car" as our example domain object. In our imagination we could store the options of a car as key/value pairs.
The options could be things like: engine type, color and so forth. <br/><br/>
The hibernate annotations that we would need to map this car entity are pretty straightforward:
<pre class="brush: java;">
@Entity
public class Car {
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
@ElementCollection
@CollectionTable(name = "options_map", joinColumns = @JoinColumn(name = "car_id"))
private Map<String, String> options = new HashMap<String, String>();
public Long getId() {
return id;
}
public Map<String, String> getOptions() {
return options;
}
}
</pre>
We created a "car" entity (and table) having a one-to-many relationship to our map storing the options as key/value pairs in map backed by a options_map table.
Now, what will happen with entries having null as value?<br/><br/>
You can argue that storing an entry with a null value does not make much sense in most scenarios.
This is probably true as the only way you would do something like this is to keep an exhaustive list of possible key values.
Even so, it would be a not so good solution since you would be duplicating your entire key set for each and every car.
A better solution would be to map the key values to enumeration instead.
Or, if the set of keys are also needed outside of your application, store them in a separate table for example.<br/><br/>
Leaving this discussion behind, the thing I want to point out here is that whichever reason you have to store entries with null as their values, think again: hibernate ignores such entries completely.<br/><br/>
Lets have a look at how hibernate handles this. In the first example we will create and store a car with three options; engine type, color and cupholder.<br/>
<pre class="brush: java;">
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = { InfrastructureContextConfiguration.class, TestDataContextConfiguration.class })
@Transactional
public class CarTest {
@PersistenceContext
private EntityManager entityManager;
@Autowired
private DataSource dataSource;
@Test
public void testWithNonNullValues() throws Exception {
// Save 3 options with non-null value
Car merc = new Car();
merc.getOptions().put("AIRCO", "DUAL-AUTO");
merc.getOptions().put("GEARBOX", "AUTO");
merc.getOptions().put("CUPHOLDER", "YES");
entityManager.persist(merc);
entityManager.flush();
entityManager.clear();
merc = entityManager.find(Car.class, merc.getId());
Assert.assertEquals(3, merc.getOptions().size());
}
}
</pre>
Nothing special happens when we run this test, the test passes: all ok.<br/><br/>
Next, we choose to drop an option (the cupholder), but we want to keep all keys existent for each car. So we will again have 3 keys, but the cupholder will have a null value: <br/>
<pre class="brush: java; highlight: [6, 13, 14]">
public void testWithNullValue() throws Exception {
// Save 3 properties, 2 with non-null value, 1 with null value
Car bmw = new Car();
bmw.getOptions().put("AIRCO", "DUAL-AUTO");
bmw.getOptions().put("GEARBOX", "AUTO");
bmw.getOptions().put("CUPHOLDER", null);
entityManager.persist(bmw);
entityManager.flush();
entityManager.clear();
bmw = entityManager.find(Car.class, bmw.getId());
// FAIL: java.lang.AssertionError: expected:<2> but was:<3>
Assert.assertEquals(3, bmw.getOptions().size());
}
</pre>
The test will fail (java.lang.AssertionError: expected:<2> but was:<3>). When we load our entity in an empty session it will only contain 2 options instead of 3. The cupholder option was removed (or simply not stored in database).<br/><br/>
This can be a problem; during the lifetime of your transaction the map will continue to have a key 'CUPHOLDER' with a null value. Only after reading data from database again (which will for example happen in a new transaction) the car loaded from database will only have two properties left, since only the properties with a non-null value were stored.<br/>
Btw; in our test case we simulate a 'new transaction' by simply flushing and clearing the session; this forces hibernate to load the data freshly again from database as would be the behaviour in a new transaction.<br/><br/>
Finally, hibernate also ignores existing records with a null value already in the database (which could be inserted by other users). In the example below we save two options (with non-null value) and then we sneakily add another option with a null value using a plain JDBC insert (we also perform a count directly afterwards to make 100% sure there are effectively three options in the database).<br/>
<pre class="brush: java;">
public void testWithNullValueInsertedByOtherUser() throws Exception {
// Save 2 options with non-null value
Car audi = new Car();
audi.getOptions().put("AIRCO", "DUAL-AUTO");
audi.getOptions().put("GEARBOX", "AUTO");
entityManager.persist(audi);
entityManager.flush();
entityManager.clear();
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
// Add another option with null value bypassing hibernate (simulating other user)
jdbcTemplate.update("insert into options_map (car_id, options_key, options) values (?, 'CUPHOLDER', null)",
audi.getId());
// Verify to make sure we now have 3 options in database
Assert.assertEquals(3,
jdbcTemplate.queryForInt("select count(*) from options_map where car_id = ?", audi.getId()));
audi = entityManager.find(Car.class, audi.getId());
// FAIL: java.lang.AssertionError: expected:<2> but was:<3>
Assert.assertEquals(3, audi.getOptions().size());
}
</pre>
The test will fail again: even though our car has 3 options in the database, hibernate will only retrieve the options with a non-null value. So instead of our expected 3 options we only get 2 options in our options map.<br/><br/>
Conclusion: during the transaction the domain model will fully reflect the data we stored in it.<br/>
If we test for the key 'CUPHOLDER' to be present it would return true.
However, in subsequent transactions, which might read the car and its options we saved before, we will only discover two options since an option with null value is not saved by hibernate and completely ignored.
This would render our operation non-idempotent if we really depended on the cupholder key to be present.<br/><br/>
To take this a bit further, if you are using Hibernate in combination with an Oracle databases you also need to be careful with empty string values.
As you are probably aware Oracle transforms an empty string (a String of length equal to 0) to a null value. So if you would have a column of type varchar2 for example, inserting '' as value will be transformed to null.<br/>
So from the Java point of view, when using Oracle, you will also "lose" properties which do not only have a null value but also have a blank string value.
<br/><br/>
It is wise to think your null/empty string strategy well through, certainly when using Hibernate in combination with the map construct.
The safest solution is probably to not depend on any entry with a blank or null value at all when using this construct.
Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com2tag:blogger.com,1999:blog-3406746278542965235.post-31675174133656578972012-07-17T09:37:00.000+02:002012-07-17T09:41:26.021+02:00Pro Spring MVC: with Web FlowToday not a technical entry, but a small announcement.<br/>
The book I wrote together with Marten Deinum is finally out!<br/>
I got my copy yesterday and it looks rather good, if I say so myself:-)<br/>
<p/>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-abiQlwZ03MQ/UAUV_lvrKvI/AAAAAAAAANw/7lJQfo5jTfQ/s1600/20120717_082555.jpg" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="400" width="280" src="http://1.bp.blogspot.com/-abiQlwZ03MQ/UAUV_lvrKvI/AAAAAAAAANw/7lJQfo5jTfQ/s400/20120717_082555.jpg" /></a></div>
<br/>
My personal contributions are the chapters on :
<ul>
<li>Testing Spring MVC applications</li>
<li>Spring Web Flow</li>
<li>Building applications with Spring Web Flow</li>
<li>Advanced Spring Web Flow</li>
<li>Spring Security</li>
</ul>
<br/>
I started writing on it around half December, the final touch (at least by us, authors) was made somewhere half June. I never expected that writing would be so difficult, it was quite a challenge. What I originally estimated as four weeks of work (two weeks full-time during vacation time and two weeks only evenings, after normal working hours) turned out to be almost five months! Writing good quality software is difficult, but writing content for a book is an even greater challenge.<p/>
At the end it was quite an interesting experience to go through the entire development cycle of a book. It was also a good learning path for the technology itself. Even though I have been working a long time with these technologies, it still took a lot of investigation to be able to write adequately about it.<p/>
Anyway, if you are doing front-end work with Spring MVC, Web Flow2 or are intrigued by these technologies, go take a look! The book is available on <a target="_blank" href="http://www.amazon.com/Pro-Spring-MVC-Professional-Apress/dp/1430241551/ref=sr_1_1?ie=UTF8&qid=1338819839&sr=8-1">Amazon</a>
or directly from <a target="_blank" href="http://www.apress.com/9781430241553">Apress</a>.<p/>
Happy reading!Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-38303005513711745902012-05-21T13:57:00.000+02:002012-07-03T13:25:44.209+02:00Migrating from Hibernate 3 to 4 with Spring integrationThis week it was time to upgrade our code base to the latest Hibernate 4.x.<br/>
We postponed our migration (still being on Hibernate 3.3) since the newer maintenance releases of the 3.x branch required some API changes which were apparently still in flux. An example is the UserType API which was still showing flaws and was going to be finalized in Hibernate 4.<br/>
<br/>
The migration went quite smooth. Adapting the UserType's to the new interface was pretty straightforward. There were some hick ups here and there but nothing painful.<br/>
<br/>
The thing to watch out for is the Spring integration. If you have been using Spring with Hibernate before, you will be using the <b>LocalSessionFactoryBean</b> (or <b>AnnotationSessionFactoryBean</b>) for creating the <b>SessionFactory</b>. For hibernate 4 there is a separate one in its own package: org.springframework.orm.<b>hibernate4</b> instead of org.springframework.orm.<b>hibernate3</b>. The <b>LocalSessionFactoryBean</b> from the hibernate 4 package will do for both mapping files as well as annotated entities, so you only need one for both flavors.<br/>
<br/>
When the upgrade was done, all our tests were running and the applications were also running fine on Tomcat using the local Hibernate transaction manager. However when running on Glassfish using JTA transactions (and Spring's <b>JtaTransactionManager</b>) we got the <b>"No Session found for current thread"</b> when calling sessionFactory.getCurrentSession();<br/>
<br/>
So it seemed I missed something in relation with the JTA configuration. As you normally do with the Spring-Hibernate integration, you let Spring drive the transactions. You specify a transaction manager and Spring makes sure all resources are registered with the transaction manager and eventually call commit or rollback. Spring will integrate with Hibernate, so it makes sure the session is flushed before a transaction commit.<br/>
<br/>
When using hibernate 3 and the hibernate 3 Spring integration, the session is bound to a thread local. This technique allow you to use the <b>sessionFactory.getCurrentSession()</b> to obtain a open session anywhere inside the active transaction. This is both the case for the local <b>HibernateTransactionManager</b> as for the <b>JtaTransactionManager</b>.<br/>
<br/>
However, as of the hibernate 4 integration, the hibernate session will be bound to the currently running JTA transaction instead.<br/>
<br/>
From a user point of view nothing changes as <b>sessionFactory.getCurrentSession()</b> will still do its job. But when running JTA this means that Hibernate must be able to lookup the transaction manager to able to register the session with the currently running transaction. This is new if you are coming from Hibernate 3 with Spring, in fact, you did not have to configure anything in regards to transactions in your Hibernate <b>SessionFactory</b> (or <b>LocalSessionFactoryBean</b>) configuration.<br/>
As it turned out, with the Hibernate 4 Spring integration the transaction manager lookup configuration is effectively done by hibernate and not by Spring's <b>LocalSessionFactoryBean</b>.<br/>
The solution was pretty simple; adding this to the Hibernate (<b>LocalSessionFactoryBean</b>) configuration solved our problems:<br/>
<br/>
<pre class="brush: xml;">
<prop key="hibernate.transaction.jta.platform">org.hibernate.service.jta.platform.internal.SunOneJtaPlatform</prop>
</pre>
<br/>
<b>"SunOneJtaPlatform"</b> should then be replaced by a subclass that reflects your container.<br/>
See the <a target="_blank" href="http://docs.jboss.org/hibernate/orm/4.1/javadocs/org/hibernate/service/jta/platform/internal/AbstractJtaPlatform.html">API docs</a> for the available subclasses.<br/>
What this class does is actually telling Hibernate how it can lookup the transaction manager for your environment. If you don't configure this there will be nothing for Hibernate to bind the session to hence throwing the exception.
<br/>
<br/>
There is also a property:
<br/>
<pre class="brush: xml;">
hibernate.current_session_context_class
</pre>
Which should point to <b>org.springframework.orm.hibernate4.SpringSessionContext</b>, but this automatically done by the <b>LocalSessionFactoryBean</b>, so there is no need to specify it in the configuration.<br/>
<br/>
As this solved my "No Session found for current thread" problem, there was still another one. Changes made to the database inside a transaction were not visible after a successful transaction commit. After some research I found out that no one was calling <b>session.flush()</b>. While with the hibernate 3 integration there was a <b>SpringSessionSynchronization</b> registered which would call <b>session.flush()</b> prior transaction commit (in the beforeCommmit method).<br/>
<br/>
In the hibernate 4 integration there is a <b>SpringFlushSynchronization</b> registered, which, as its name says, will perform a flush also. However, this is only implemented in the actual “flush” method of the <b>TransactionSynchronization</b>, and this method gets never called.<br/>
<br/>
I raised an <a target="_blank" href="https://jira.springsource.org/browse/SPR-9404">issue</a> for this on Spring bugtracker, including two sample application which illustrates the problem clearly.<br/>
The first uses Hibernate 3 and the other is the exact same application but this time using hibernte 4. The second will show that no information is actually persisted to database (both apps are tested under the latest Glassfish 3.1.2)<br/>
<br/>
Until now the best workaround seems to be creating a flushing Aspect that wraps around <b>@Transactional</b> annotations. Using the order attribute you can order the transactional annotation to be applied before your flushing Aspect. This way your Aspect is still running inside the transaction, and is able to flush the session. It can obtain the session the normal way by injecting the <b>SessionFactory</b> (one way or the other) and then calling <b>sessionFactory.getCurrentSession().flush()</b>.
<pre class="brush: xml;">
<tx:annotation-driven order="1"/>
<bean id="flushinAspect" clas="...">
<property name="order" value="2"/>
</bean>
</pre>
<br/>or, if using the annotation configuration:<br/>
<pre class="brush: java;">
@EnableTransactionManagement(order=1)
</pre>
<br/>
<b><u>Update:</u></b><br/>
There was some feedback on the issue. As it turns out it does not seem to be a bug in the Spring Hibernate integration, but a missing Hibernate configuration element. Apparently the "hibernate.transaction.factory_class" needs to be set to JTA, the default is JDBC which depends on the Hibernate Transaction API for explicit transaction management. By setting this to JTA the necessary synchronizations are registered by hibernate which will perform the flush. See the Spring <a target="_blank" href="https://jira.springsource.org/browse/SPR-9404">https://jira.springsource.org/browse/SPR-9404</a>
<p/>
<b><u>Update 2:</u></b><br/>
As it turns out, after correcting the configuration as proposed on the preceding issue, there was still a problem. I'm not going to repeat everything, you can find detailed information in the second bug entry I submitted here: <a target="_blank" href="https://jira.springsource.org/browse/SPR-9480">https://jira.springsource.org/browse/SPR-9480</a><br/>
It basically comes down to the fact that in a JTA scenario with the JtaTransactionFactory configured, hibernate does not detect that it is in a transaction and will therefore not execute intermediate flushes. With the JtaTransactionFactory configured, you are expected to control the transaction via the Hibernate API rather then via an external (Spring in our case) mechanism. One of the side effects is that you might be reading stale data in some cases.<p/>
Example:
<pre class="brush: java;">
//[START TX1]
Query query = session.createQuery("from Person p where p.firstName = :firstName and p.lastName = :lastName");
Person johnDoe = (Person)query.setString("firstName","john").setString("lastName","doe").uniqueResult();
johnDoe.setFirstName("Jim");
Person jimDoe = (Person)query.setString("firstName","jim").setString("lastName","doe").uniqueResult();
//[END TX1]
</pre/>
What happens is that when performing the second query at line 5, hibernate should detect that it should flush the previous update which was made to the attached entity on line 4 (updating the name from 'john' to 'jim').<p/>
However, because hibernate is not aware it is running inside an active transaction, the intermediate flushing doesn't work. It will only flush once before the transaction commits. This results in stale data, as the 2nd query would not find "jim" and return null instead. The solution (see the reply from Juergen Hoeller in the issue) is configuring <b>hibernate.transaction.factory_class</b> to <b>org.hibernate.transaction.CMTTransactionFactory</b> instead. At first I was a bit sceptical, as CMT makes be thing about EJB containers. However, if you read the Java doc on CMTTransaction it does make sense:
<pre class="brush: java;">
/**
* Implements a transaction strategy for Container Managed Transaction (CMT) scenarios. All work is done in
* the context of the container managed transaction.
*
* The term 'CMT' is potentially misleading; the pertinent point simply being that the transactions are being
* managed by something other than the Hibernate transaction mechanism.
*
* Additionally, this strategy does *not* attempt to access or use the {@link javax.transaction.UserTransaction} since
* in the actual case CMT access to the {@link javax.transaction.UserTransaction} is explicitly disallowed. Instead
* we use the JTA {@link javax.transaction.Transaction} object obtained from the {@link TransactionManager}
</pre>
<p/>
After that everything seems to work fine. So to conclude, if you want hibernate to manage the JTA transaction via the UserTransaction, you should use JtaTransactionFactory. In that case you must use the Hibernate API to control the transaction. If there is someone else managing the transaction (Spring, EJB container ...) you should use CMTTransactionFactory instead. Hibernate will then revert to registering synchronisations, by checking for active javax.transaction.Transaction using the javax.transaction.TransactionManager. If there is any other issue popping up I'll update this entry accordingly.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com4tag:blogger.com,1999:blog-3406746278542965235.post-79511418640969023612011-10-07T15:48:00.000+02:002013-08-02T12:35:50.964+02:00Oracle AQ with SpringIn my current project we use a Java SEDA. The MOM to support this is IBM (Websphere) MQ. Our most used object is the queue, which enables us to handle events asynchronously and by multiple consumers which greatly improves scalability and robustness.<br />
<br />
From a Java point of view the MOM implementation is really not that important, as it is accessed via the JMS API. So whether its Websphere MQ, JBoss MQ, ... as long as it has JMS support its pretty transparent. We do use some minor MQ specific extensions (PCF) to get queue depths and the likes, but that is more from an operational management point of view.<br />
<br />
The choice of MQ was made before I joined the project, probably because other legacy subsystems have the least trouble dealing with MQ since they are already IBM based. Although we don't benefit a lot from it possibilities, since there is no QueueManager to QueueManager communication or the likes in which MQ is really strong. But it has to be said that MQ is a solid and mature product with a lot of possibilities.<br />
<br />
The downside is probably its price (especially if you under-use it) and that it requires specific MQ knowledge to operate and maintain a running instance. For example; moving messages from a queue to another natively on Solaris is not a trivial thing if your not into the MQ administration (no, the 'MOVE' command is not supported on MQ Solaris).<br />
<br />
Since we are using 2 resources most of the time, this also implies that our backends are running XA transactions to make 2PC work between our MOM and RDBMS (Oracle).<br />
A while ago someone threw the idea on the table to switch to Oracle AQ (Advanced Queues) which is the Oracle MOM implementation. I'm not going in the area of comparing MQ vs AQ, but the fact is that AQ supports JMS and is a fully fledged MOM (It also has Topics, btw), so on paper it is more then enough for our usages.<br />
<br />
Cool detail is that JMS Connection that you obtain is actually backed by a normal (JDBC) database connection. In fact, what happens is that the AQ driver uses a datasource under the hood. If you do a Queue.publish() or a Queue.read() the AQ driver will translate that to stored procedure and send them through the SQL datasource you instantiated it with. This also means we could drop our XA, since we only need to enlist a single resource for both our MOM and RDBMS access.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-igbOibtl-zY/To76MY9huPI/AAAAAAAAAMU/WduhbEO-OA0/s1600/oracleaq.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="272" width="400" src="http://3.bp.blogspot.com/-igbOibtl-zY/To76MY9huPI/AAAAAAAAAMU/WduhbEO-OA0/s400/oracleaq.png" /></a></div><br />
To set this up my first idea was to look for a resource adapter (RAR) which would enable AQ via the application server (Webshere MQ also ships with a JEE RAR). At that point I did not knew how it would handle the JDBC connection sharing if connection would be made via the RAR, but anyway. Quickly I found out that there is no real AQ resource adapter available for other JEE servers then Oracle AS itself (for this I was using Glassfish btw).<br />
<br />
There is: <a target="_blank" href="http://genericjmsra.java.net/">genericjmsra</a> but you cannot use it "properties based" like you enter the uri/username/password of the MOM. See <a target="_blank" href="http://genericjmsra.java.net/docs/oracleaq/glassfish-oracleaq.pdf">here</a> for its AQ specific manual:<br />
<br />
Quote:<br />
<i><br />
Oracle JMS client does not allow creation of ConnectionFactory, QueueConnectionFactory or<br />
TopicConnectionFactory utilizing JavaBean approach. The factory creation is only possible through AQjmsFactory class provided in the Oracle jms client api. However fortunately, Oracle does support the JNDI lookup approach. We will be focusing on the JNDI approach for Oracle AQ and glassfish integration<br />
</i><br />
This means you need an Oracle LDAP server in which some remote objects are published which are then looked up by the RA. So sharing the same JDBC connection for relational access and AQ will certainly not be possible this way.<br />
<br />
Fortunately you can use the AQJmsFactory (that's the main factory which you feed a datasource and it gives you back a JMS ConnectionFactory) directly from your code, but that would require some boiler plate code as the AQJmsFactory checks that the actual connection is a direct Oracle connection.<br />
<br />
If you are using a JDBC pool, like for example C3PO or Commons DBCP they will wrap the connections (in order to suppress closes etc) and these connections will be rejected because they are no direct instance of the Oracle connection. Thankfully a new Spring module was released at the right time and comes to the rescue with: <a target="_blank" href="http://www.springsource.org/spring-data/jdbc-extensions">Spring jdbc-extensions</a>.<br />
<br />
This is that boiler plate you want to seamlessly integrate Oracle AQ with your existing Spring managed datasource and transactions. The extension will make sure the Oracle AQJmsFactory is given a proxy which will be an instance of the Oracle connection. The proxy enables us to control what we give the Oracle AQ implementation. <br />
<br />
For example when it tries to call 'close' we will suppress the call, since we know it will be handled by transaction manager (datasource,hibernate, jta, ...) later on. If your interested in this check the source at: org.springframework.data.jdbc.config.oracle.AqJmsFactoryBeanFactory. <br />
That is the custom namespace handler for the AQ Spring XML config which creates the appropriate beans to do the boiler plate.<br />
<br />
In this first example we create a scenario in which an event is received (Q1), a database record is inserted (T1) and a second event is published (Q2). All of this should run in one transaction, so if there is a failure at any point everything should be reverted (1 message back on Q1, no records in T1, and no messages on Q2). If everything succeeds, the message from Q1 should be processed, the record inserted and a new message published on Q2.<br />
<br />
To start I'm going to setup the two AQ queue's and their queue table:<br />
<br />
<pre class="brush: sql;">EXECUTE DBMS_AQADM.CREATE_QUEUE_TABLE(queue_table => 'Q1_T', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE');
EXECUTE DBMS_AQADM.CREATE_QUEUE (Queue_name => 'Q1', Queue_table => 'Q1_T', max_retries => 2147483647);
EXECUTE DBMS_AQADM.START_QUEUE (Queue_name => 'Q1');
EXECUTE DBMS_AQADM.CREATE_QUEUE_TABLE(queue_table => 'Q2_T', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE');
EXECUTE DBMS_AQADM.CREATE_QUEUE (Queue_name => 'Q2', Queue_table => 'Q2_T', max_retries => 2147483647);
EXECUTE DBMS_AQADM.START_QUEUE (Queue_name => 'Q2');
</pre><br />
On AQ each Queue needs to have a corresponding queue table. The queue table is the table where the data is physically stored. You will never talk to a queue table directly, but you can use it with DML to query them via your favorite database IDE. On each you can specifiy additional properties, on the queue table you have to specifiy which payload it will have. On the queue itself you can specifiy after how many unsuccesful dequeues the message is moved to the exception queue. <br />
<br />
In our project we make use of an application level failover and DLQ management system with separate queueing. So we don't need this feature. There is however no way to turn this off, so we've chosen the max setting (which is Integer.MAX_VALUE). Btw; the exception queues are generated automatically, you have no control over them.<br />
<br />
To check if everything is created:<br />
<br />
<pre class="brush: sql;">select * from all_queues where name like 'Q1%' or name like 'AQ$_Q1%' or name like 'Q2%' or name like 'AQ$_Q2%'
</pre>The results:<br />
<br />
<div align="center"><table border="1" cellpadding="0" cellspacing="0"><tr> <th>NAME</th> <th>QUEUE_TABLE</th> <th>QID</th> <th>QUEUE_TYPE</th> <th>MAX_RETRIES</th> </tr>
<tbody id="data">
<tr> <td>Q2</td> <td>Q2_T</td> <td align="right">365831</td> <td>NORMAL_QUEUE</td> <td align="right">2147483647</td> </tr>
<tr> <td>AQ$_Q2_T_E</td> <td>Q2_T</td> <td align="right">365830</td> <td>EXCEPTION_QUEUE</td> <td align="right">0</td> </tr>
<tr> <td>Q1</td> <td>Q1_T</td> <td align="right">365816</td> <td>NORMAL_QUEUE</td> <td align="right">2147483647</td> </tr>
<tr> <td>AQ$_Q1_T_E</td> <td>Q1_T</td> <td align="right">365815</td> <td>EXCEPTION_QUEUE</td> <td align="right">0</td> </tr>
</tbody></table></div><br />
Next we'll setup our Spring config. The goal is to create a message consumer that listens for messages on Q1 and processes them. Our processing will consist of inserting a record in T1 and putting a message on Q2.<br />
<pre class="brush: xml; highlight: [23, 37];"> <!-- Sets up the JMS ConnectionFactory, in this case backed by Oracle AQ -->
<bean id="oracleNativeJdbcExtractor" class="org.springframework.jdbc.support.nativejdbc.SimpleNativeJdbcExtractor"/>
<orcl:aq-jms-connection-factory id="connectionFactory" data-source="dataSource" use-local-data-source-transaction="true" native-jdbc-extractor="oracleNativeJdbcExtractor"/>
<tx:annotation-driven/>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" lazy-init="true">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:host:port:SID"/>
<property name="username" value="Scott"/>
<property name="password" value="Tiger"/>
</bean>
<!-- Using DataSourceTxManager, but could also be HibernateTxManager or JtaTxManager -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" lazy-init="true">
<property name="dataSource" ref="dataSource"/>
</bean>
<!-- You can also construct the JMSTemplate in code, but we'll do it here so its all together in one place -->
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="defaultDestinationName" value="Q2"/>
<property name="sessionTransacted" value="true"/>
</bean>
<bean id="myMessageListener" class="be.error.jms.MyMessageListener">
<property name="dataSource" ref="dataSource"/>
<property name="jmsTemplate" ref="jmsTemplate"/>
</bean>
<!-- Once it is started, it will try to read messages from Q1 and let 'messageListener' process them -->
<bean id="messageListenerContainer"class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="transactionManager" ref="transactionManager"/>
<property name="destinationName" value="Q1"/>
<property name="messageListener" ref="myMessageListener"/>
<property name="sessionTransacted" value="true"/>
</bean>
</pre>As you can see the magic is in the orcl:aq-jms-connection-factory which will make a JmsConnectionFactory available under the id 'connectionFactory' and using our datasource to do the AQ queueing.<br />
<br />
<b>Very important:</b> if you don't want to spend half a day investigating some weird transaction behaviour (I even mistakenly thought it was a bug and pointed that out <a target="_blank" href="https://jira.springsource.org/browse/DATAJDBC-9">here</a>) I suggest to read this: <br />
<br />
In my configuration you will see that the 'sessionTransacted' is set to "true" for the JmsTemplate and for the DefaultMessageListenerContainer. This makes sense as we are running outside of a JEE managed environment and we want to have local transactions for our JMS operations. The theory behind it is however a bit more complex.<br />
<br />
When running outside of a JEE managed environment you have the choice of letting your session interaction be part of a local transaction. This is controlled by the sessionTransacted setting (it maps directly on the JMS API). This means that if you consume messages from different objects belonging to the same session, they will be controlled in a single transaction.<br />
For example, I create QueueSession #1 and I use it to consume a message from Q1 and consume another message from Q2. After consuming both messages, I can issue a session.rollback() and everything is brought back to its initial state. If I would have used no transactions, I would be working with an acknowledgement mode. Suppose I would have chosen CLIENT_ACKNOWLEDGE then I had to acknowledge on message level whether my message was successfully consumed. So I would have first retrieved message #1 from Q1 and then message #2 to Q2 (all via QueueSession #1). In the end I would have to do:<br />
<br />
<pre class="brush: java;">messageOne.acknowledge();
//system crashes here
messageTwo.acknowledge();
</pre>This could of course create inconsistency as in my example messageOne was marked consumed but messageTwo wasn't. This is only a problem if your unit of work should be treated in an atomic way. If it is you should use at least local transactions.<br />
<br />
When you want to consume/produce messages from a Queue and do interaction with another resource (RDBMS) for example you should use a distributed transactionmanager (in our case that would mean JTA). But remember that we are not dealing with different resources here, it all comes down to a single database connection. So in our case the "local transaction" is a bit "longer local" then it would normally be as it also includes all our (SQL) calls made to that same database connection as the JMS infrastructure is using.<br />
<br />
In our case the DataSourceTransactionManager will control the local transaction, and that includes JMS operations as well as SQL operations issued via JDBC. It is that component which will call commit or rollback. there is no need for intermediate commits on the queueSession. <br />
<br />
So basically: by setting sessionTransacted to true, no one performs intermediate commits and leaves everything to whoever controls the transaction, in our case DataSourceTransactionManager.<br />
Make sure you use JdbcTemplate for direct JDBC access and JmsTemplate for MOM access. Make sure sessionTransacted is set to true when you should create JmsTemplate in code. Also, the DefaultMessageListenerContainer is a JMS receiver and must also be sessionTransacted for the same reason.<br />
<br />
You might want to be tempted to remove the sessionTransacted from the JmsTemplate and DefaultMessageListenerContainer if you are running in an JEE environment. The JMS API says that the values to sessionTransacted and acknowledgementMode are ignored in such case.<br />
While this is true in general, it is not true in this case. The Oracle AQ will not properly detected that it is running in a JEE JTA environment if you are using anything else then Oracle AS. If you remove the property, then the driver will perform intermediate commits and your transaction will be broken. So also in JEE mode you will have to leave this set to true!<br />
<br />
But don't worry, in the JTA case your datasource will then be XA enabled and the transactionmaanger performing commmits will be the JtaTranasctionManager. As far as the AQ driver is concerned it sees no difference (all transaction handling an coordination is done at an higher level).<br />
<br />
Also, I'm using a DataSourceTransactionManager here, since I only require direct JDBC access.<br />
If you would be using hibernate, you could use HibernateTransactionManager. You could then do AQ, plain JDBC access and work with hibernate's SessionFactory at the same time.<br />
If you would still have another resource (maybe a 2nd RDBMS) and still want XA, you can simply plugin the JTA transaction manager without any problem (its just a matter of switching configuration).<br />
<br />
For the Java messageListener part, this is all standard:<br />
<br />
<pre class="brush: java;">public class MyMessageListener implements SessionAwareMessageListener<Message> {
private DataSource dataSource;
private JmsTemplate jmsTemplate;
@Override
public void onMessage(Message message, Session session) throws JMSException {
//Message received from Q1 via 'messageListenerContainer'
TextMessage textMessage = (TextMessage) message;
System.out.println("Received message with content:" + textMessage.getText());
//Insert its content into T1
new JdbcTemplate(dataSource).update("insert into T1 values (?)", textMessage.getText());
System.out.println("Inserted into table T1");
//Publish a message to Q2
jmsTemplate.send(new MessageCreator() {
@Override
public Message createMessage(Session session) throws JMSException {
TextMessage textMessage = session.createTextMessage();
textMessage.setText("<somexml>payload</somexml>");
return textMessage;
}
});
System.out.println("Sended message to Q2");
}
public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
}
public void setJmsTemplate(JmsTemplate jmsTemplate) {
this.jmsTemplate = jmsTemplate;
}
}
</pre>I then created a small forever blocking test case to quickly fire up the application context so that the DefaultMessageListenerContainer could start looking for messages on Q1. <br />
<br />
<pre class="brush: java;">@Test
@ContextConfiguration(locations = { "classpath:/spring/aq-test.xml" })
public class OracleAqTransactionResourceTest extends AbstractTestNGSpringContextTests {
@Autowired
private DataSource dataSource;
private JdbcTemplate jdbcTemplate;
@BeforeMethod
public void setup() {
jdbcTemplate = new JdbcTemplate(dataSource);
}
public void testSingleTransaction() {
System.out.println("Running...");
blockUntillReadyOrTimeout();
System.out.println("Done.");
}
private void blockUntillReadyOrTimeout() {
while (true) {
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
}
</pre>After launching the test, I inject a message into Q1 (I use Oracle SQL developer):<br />
<br />
<pre class="brush: sql;">DECLARE
msg SYS.AQ$_JMS_TEXT_MESSAGE;
queue_options DBMS_AQ.ENQUEUE_OPTIONS_T;
message_properties DBMS_AQ.MESSAGE_PROPERTIES_T;
message_id RAW(30);
BEGIN
msg := SYS.AQ$_JMS_TEXT_MESSAGE.CONSTRUCT();
msg.set_text('testing 123');
DBMS_AQ.ENQUEUE(
queue_name => 'Q1',
enqueue_options => queue_options,
message_properties => message_properties,
payload => msg,
msgid => message_id);
commit;
END;
</pre>And off we go:<br />
<br />
<pre>Running...
Received message with content:testing 123
Inserted into table T1
Sended message to Q2
</pre>In oracle we see that the message is present on Q2 (at least its queue table):<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Yoj1woqSRBc/To8CrgvmajI/AAAAAAAAAM0/ZCd4oVULCbo/s1600/sqldev1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="219" width="400" src="http://2.bp.blogspot.com/-Yoj1woqSRBc/To8CrgvmajI/AAAAAAAAAM0/ZCd4oVULCbo/s400/sqldev1.png" /></a></div>And that a record is inserted into T1:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-tDZ5LWwpz9w/To8C04x6EiI/AAAAAAAAAM8/pdIWiF6O8a8/s1600/sqldev2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="292" width="400" src="http://3.bp.blogspot.com/-tDZ5LWwpz9w/To8C04x6EiI/AAAAAAAAAM8/pdIWiF6O8a8/s400/sqldev2.png" /></a></div>You are free to play with some transaction scenario's, as creating multiple (possibly nested) transactions, let them rollback etc. I performed 5 scenario's and they all worked fine.<br />
<br />
PS. make sure you use at least spring-jdbc 1.0_M2 (or up) since we discovered a <a target="_blank" href="https://jira.springsource.org/browse/DATAJDBC-8">small bug</a> in M1 which could cost you some time to investigate :)Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com18tag:blogger.com,1999:blog-3406746278542965235.post-48412090489585858232011-07-12T17:45:00.000+02:002011-07-13T08:36:16.206+02:00webOS app development for the Pre2 with JAX-RS and AtmosphereRecently I tried the development possibilities for my Palm Pre2. As it goes, this smartphone runs some modified linux distro (aka webOS). By default you can only access the 'window manager': the graphical UI which makes your phone, well, 'your phone'.<br />
<br />
So the first thing you need to do if you want to get deeper into the OS is enabling developer mode. This mode allows you to install custom applications, gain console access and the likes.<br />
To put the phone in developer mode, see here at the bottom of the page: <a href="https://developer.palm.com/content/index.php?id=4522" target="_blank">webOS 2.0 Devices</a><br />
<br />
The first thing I tried after enabling developer mode was <a target="_blank" href="http://preware.org/#/index/">preware</a>. From their <a target="_blank" href="http://www.webos-internals.org/wiki/Application:Preware#Key_Features">wiki</a>:<br />
<br />
<i>Preware is a package management application for the Palm Pre and the Palm Pixi. <br />
Preware allows the user to install any package from any of the open standard package repositories on preware.org (or any other location that hosts an open standard package repository). <br />
Preware relies on a custom written service developed from community research which allows the mojo app to talk to the built-in ipkg tool. <br />
</i><br />
Do note that this is a community product and is not part of the 'official SDK' neither do you need it for development. You can basically use it to:<br />
<ul><li>Install/deinstall apps from open repositories (a lot of apps are already available via the preware repo)</li>
<li>Install updates/check phone info</li>
<li>Gain terminal access to your pre</li>
<li>Install the novacom driver, which will also be required for the SDK (preware can install it automatically for you)</li>
<li>Probably more, see their site</li>
</ul><br />
Since I was going to try this on Linux (Ubuntu Maverick), I was already preparing for a long night of debugging, catting var/log/messages, resolving and finding missing dependencies, compiling my kernel for some support that would be missing, searching the web for usb problems and so forth.However, as it turned out, none of this was required (no, it really wasn't). Basically, you first enable developer mode, download preware, extract it, plug your Pre2 into the USB (select 'just charge' on the phone) and run preware:<br />
<br />
<pre class="brush: shell;">~/Desktop/Palm Pre 2$ java -jar WebOSQuickInstall-4.2.0.jar
</pre><br />
It will ask you to install novacom on first run and after that you'll get the main screen. <br />
Launching the terminal looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-c5jLflt7zCM/ThxlRyzhzdI/AAAAAAAAAL4/SHUwUdE-oSI/s1600/console.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="258" width="400" src="http://3.bp.blogspot.com/-c5jLflt7zCM/ThxlRyzhzdI/AAAAAAAAAL4/SHUwUdE-oSI/s400/console.JPG" /></a></div><br />
I used preware to access the console of the phone linux distro via the build in terminal. After that you can probably install or configure an SSH server on the phone. I did not explore this route any further since I only have one phone and I don't want a lot of services draining my battery, lulzers hacking my phone, or needing to hard reset my phone every week.<br />
<br />
Secondly I also installed a terminal app on the phone itself via preware.Currently I have problems in my car when its connected via bluetooth. I can see the phonebook, the network connection etc, but I cannot make phone calls. So when next time I'm stuck traffic I'm planning to do some on the fly debugging to find out whats wrong. The terminal app I installed was 'SDLTerminal', the other ones; terminal and terminus didn't work for me (at least not on a install/run lazy fashion)<br />
<br />
What the SDK is concerned, it is really easy and it works perfect on my ubuntu. The steps can be found <a target="_blank" href="https://developer.palm.com/content/resources/develop/developing_with_the_eclipse_ide.html">here</a>. As Java developer I'm using it via eclipse, although other possibilities exist, see their site.<br />
<br />
Summarized:<br />
<ul><li>Get the latest Eclipse, they advice you to use the Web developer profile</li>
<li>Install the webOs plugin via eclipse</li>
<li>Install the Aptana plugin via eclipse (optional)</li>
<li>Restart eclipse</li>
</ul><br />
<a target="_blank" href="https://developer.palm.com/index.php?option=com_content&view=article&layout=page&id=1788#">Install the SDK</a> I skipped: Java (allready had Java6), ia32-libs (already installed for some other 32bit comatibility I required), novacom (already installed by preware). I also installed the latest version of virtualbox (v4), and then removed it again to install 3.2, since the SDK must have a virtualbox version: >=3.0 && <=3.2.
Now, after you type:
<pre class="brush: shell;">palm-emulator </pre>You'll see virtualbox popping up and booting webOS (the image came along with the SDK):<br />
<table><tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-3wXXy137_hE/ThxmfSWqwWI/AAAAAAAAAMA/z22CaSawl1M/s1600/vbox1.JPG" imageanchor="1" style=""><img border="0" height="400" width="230" src="http://2.bp.blogspot.com/-3wXXy137_hE/ThxmfSWqwWI/AAAAAAAAAMA/z22CaSawl1M/s400/vbox1.JPG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-V5Bm2llCoKk/Thxmmwf2zOI/AAAAAAAAAMI/UlWvylwEGSU/s1600/vbox2.JPG" imageanchor="1" style=""><img border="0" height="400" width="230" src="http://4.bp.blogspot.com/-V5Bm2llCoKk/Thxmmwf2zOI/AAAAAAAAAMI/UlWvylwEGSU/s400/vbox2.JPG" /></a></div></td> </tr>
</table><br />
Virtualbox takes your NIC into account, and sets up a NAT connection. The latter means that your emulated webOS can go out on your network; packets get the source address from your host nic, but does not have an ip on your network. So by default you won't be able to access your webos vm FROM the network. If you want that, the easiest thing is probably to switch the network adapter configuration in virtualbox to "bridged" instead (or setup portforwarding).<br />
<br />
To get a console to the webOS vm, you have two options: <br />
<ul><li>The novacom driver acts as middleware both for you phone connected via USB as well as webOS running in the VM. So you can access the console of the vm webOS via the terminal option in preware, just as you would to to access it on the pre.</li>
<li>Even though NAT is used by default, there are some port mappings made in the virtualbox image configuration. This means that certain ports on the host are forwarded by virtualbox to the VM, which does allow you to connect to the vm without any extra setup when using NAT. But only for these ports, and only from your machine. For example port 5522 is mapped to the webOS VM on 22.</li>
</ul>Do note that latter is a difference with the 'production' webOS image on the pre2: there is no such ssh daemon running by default, as I addressed previously. There are other differences between the vm and the one on the phone; like no camera, gravity meter etc. For example, ssh'ing to your webos VM; <pre class="brush: shell;">ssh root@localhost -p 5522
</pre>(There is no password, just hit enter)<br />
<br />
Next I followed <a target="_blank" href="https://developer.palm.com/content/resources/develop/building_your_first_app.html">this howto</a> to create my first app. If you are lazy, you could just: <ol><li>Startup the emulator (commandline: 'palm-emulator')</li>
<li>Startup your webos-plugin-enabled eclipse</li>
<li>File->New->Palm webOS->Hello World application</li>
<li>Run->Run</li>
<li>Look on your virtualbox webOS VM, you'll see your application running</li>
</ol>To run the appp on your pre2 instead, just plugin the pre2 in the USB, select 'just charge' and <ol><li>Run->Run</li>
<li>Look on your pre2</li>
</ol>Finally, I was eager to find out how hard it is to extend the hello world app and let it do something cool. Following the moto 'go hard or go home', my idea was to extend the example to send a photo (either taken directly or selected from FS) in JSON format to a RESTful webservice via ajax. There would also needed to be a web page that queries the same webservice using Ajax push (Comet). So, the flow should look like this: <ol><li>Startup the app in webOS</li>
<li>Click a button which brings the user to a select/take picture menu</li>
<li>Select (or select freshly taken) picture</li>
<li>In the app the picture thumbnail and the path on the device should be shown</li>
<li>After hitting a send button, the image is send to the RESTful webservice</li>
<li>A possible browser window connected to the server will get the newly uploaded picture automatically pushed</li>
</ol>To do this I first created a small RESTful webservice via JAX-RS. I used Athmosphere to do the comet part. <pre class="brush: java;">@Path("upload")
@Singleton
public class ImageService {
private static final String THE_IMAGE_LOC = "/tmp/theimage.jpg";
private Broadcaster topic = new JerseyBroadcaster();
@POST
@Consumes(MediaType.APPLICATION_JSON)
@Broadcast
public Broadcastable uploadImage(FileContent fileContent)
throws IOException {
FileUtils.writeByteArrayToFile(new File(THE_IMAGE_LOC),
Base64.decodeBase64(fileContent.getData()));
return new Broadcastable(fileContent.getData() + "END", "", topic);
}
@GET
public SuspendResponse<string> suspend() {
return new SuspendResponse.SuspendResponseBuilder<string>()
.broadcaster(topic).outputComments(false).build();
}
}
</pre>The web.xml:<br />
<pre class="brush: xml;"><servlet>
<description>AtmosphereServlet</description>
<servlet-name>AtmosphereServlet</servlet-name>
<servlet-class>org.atmosphere.cpr.AtmosphereServlet</servlet-class>
<init-param>
<param-name>org.atmosphere.core.servlet-mapping</param-name>
<param-value>/ImageService</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>AtmosphereServlet</servlet-name>
<url-pattern>/ImageService/*</url-pattern>
</servlet-mapping>
</web-app>
</pre>The browser will call the 'suspend' method which will block on the topic untill new data is available. When an image is uploaded, the broadcast will trigger the suspend method to resume for possible connected clients. In the result they'll find the image in base64 format. Atmosphere also comes with a JQuery based plugin, which makes it easy to make an Ajax connection to the webservice. Via the plugin you can easily switch between comet types or even websocket. In this case I've used long polling. <pre class="brush: javascript;"><script type="text/javascript">
$(document).ready(function() {
var callbackAdded = false;
var lastRead=0;
function subscribe() {
function callback(response) {
var data = response.responseBody;
if(data.length > 3 && data.substring(data.length - 3, data.length) == "END"){
data = data.substring(lastRead, (data.length-3));
lastRead=lastRead+data.length+3;
$("#imageHolder").replaceWith("<img width='"+ ( $(window).height())+"' height='"+( $(window).height()/1.2) +"' id='imageHolder' src='data:image/jpg;base64,"+data+"'/>");
}
}
var location = "ImageService/upload";
$.atmosphere.subscribe(location, !callbackAdded ? callback : null, $.atmosphere.request = { transport: "polling" });
callbackAdded = true;
}
subscribe();
});
</script>
</pre>As you can see I directly injected the base64 string in an HTML img tag. I also applied some superior scaling algorithm to make the images a bit smaller.<br />
<br />
At this point, there were some problems: <ul><li>On my setup this was not doing long polling, but streaming. In case of long polling a new connection is made once data has been returned from the server.<br />
In my case the connection remained open forever (==streaming). Maybe I need some server side config for long polling? I could not find this anywhere...</li>
<li>Every time the callback is called, you get the newly written data AND the previously written data back.<br />
So when sending a second image, the response still contains the first image. My guess is that this is a side effect of streaming?<br />
To get around this I'm keeping a 'lastRead' variable as you can see.</li>
<li>The callback is called multiple times for different chunks of the received data. This means that I must find a way to denode the end of a transmission.<br />
Apparently there are some characters added to the end (CRLF I assume) so I have could used these, but to lazy to find out, so I added a token myself (namely "END").<br />
</li>
</ul>For the JS imports, this is all I used: <pre class="brush: javascript;"><script src="js/jquery-1.4.2.js" type="text/javascript"></script>
<script src="js/jquery.atmosphere.js" type="text/javascript"></script>
</pre>The atmosphere plugin came bundled with JQuery 1.4.2 so I used that version, no idea if it also works with newer ones. Btw the jquery.atmosphere.js plugin I found it <a target="_blank" href="https://oss.sonatype.org/content/repositories/releases/org/atmosphere/atmosphere-jquery/">here</a> (for the download link of this app see below).<br />
<br />
Now, for the Pre2 application: the applications that plugin into webOS are written in Javascript (yes, that 0 == "" language). For the record, you can also develop them in C/C++ if you need speed and very low level access to the hardware. The developer site explains (in a very annoying and hard to use layout) that there is a broad javascript platform which contains all the components you need to build applications fast. I was quite impressed by the amount of components/services/widgets available, it is like a JQuery/prototype on steroids. There is even test support built in. If you know JQuery or prototype a lot of these things will seem familliar. <br />
<br />
All of this allows you to develop via javascript on a higher level. I knew the Ajax thing was going to be easy, so the first thing I tried out was how to read a (binary) file and transform it to base64. Apparently webOS (2+) uses "Node.js", so thats piece of cake. The hard part was to learn that you can only access the node functionality from a headless service rather then an application (not 100% sure on this, but thats how it looked like). The next annoying thing to find out was that I could not get the eclipse plugin to package my service along with my app. So I have to manually package and upload the service to the phone. The service code looks like this: <pre class="brush: javascript;">var fs = IMPORTS.require('fs');
var ReadFileAssistant = function(){
}
ReadFileAssistant.prototype.run = function(future) {
var file = fs.readFileSync(this.controller.args.filePath);
var fileBase = file.toString("base64",0,file.length)
future.result = { reply: fileBase};
}
</pre>I used a hello world service sample (from the SDK docs&samples), removed the hello world stuff and added my code in place. The "buildpackage" file contains the commands you need to build an ipk from it and get that on your phone/emulator. You can run the service, but that is not required. As it is headless it won't show you anything, besides the entry page of the application. From the moment it is installed you'll be able to access the service from any application (at least when it is made "public"). Btw; apparently a service must be package inside an application. <br />
<br />
So in the end I will have two applications: one with the service inside and one with the UI. But this is only because eclipse withheld me from packing them together (and packing the UI manually together with the service would take to much time for testing).<br />
<br />
Next we have the UI part which is made out of HTML: <pre class="brush: html;"><div id="main" class="palm-hasheader">
<div class="palm-header">Demo app</div>
<div x-mojo-element="Drawer" id="drawerId" class="drawerClass" name="drawerName">
<div class="palm-group">
<div class="palm-group-title" x-mojo-loc=''>Image:</div>
<div class="palm-list">
<div class="first row">
<div class="palm-row-wrapper textfield-group" x-mojo-focus-highlight="true">
<div x-mojo-element="ImageView" id="ImageId" class="ImageClass" name="ImageName" align="center"></div>
</div>
</div>
<div class="last row">
<div class="palm-row-wrapper">
<div id="File" style="font-size: 10px" x-mojo-element="TextField" />
</div>
</div>
</div>
</div>
</div>
</div>
<div id="Select" name="Select" x-mojo-element="Button"></div>
<div id="Send" name="Send" x-mojo-element="Button"></div>
<div id="result" class="palm-body-text" />
</div>
</pre>The "assistent" then, which forms the logic behind the view: <pre class="brush: javascript;">FirstAssistant.prototype.handleSelect = function(event) {
var self = this;
var params = {
defaultKind : 'image',
onSelect : function(file) {
self.controller.get('File').innerHTML = file.fullPath;
self.controller.get('ImageId').mojo.centerUrlProvided(file.fullPath);
self.controller.get('drawerId').mojo.setOpenState("true");
}
};
Mojo.FilePicker.pickFile(params, this.controller.stageController);
}
FirstAssistant.prototype.handleSend = function(event) {
var that = this;
var filePath = that.controller.get('File').innerHTML;
this.controller.serviceRequest("palm://com.palmdts.servicesample.service",
{
method : "readFile",
parameters : {
"filePath" : filePath
},
onSuccess : function(response) {
var contentToSend = {filePath:filePath, data: response.reply};
$.ajax({
type : "POST",
contentType:"application/json",
url : "http://192.168.0.1:8080/WebOSServer/ImageService/upload",
data: Object.toJSON(contentToSend),
cache : false,
success : function(result) {
that.controller.get("result").innerHTML = "File Sent!<br/>"+ result;
},
error: function(result){
that.controller.get("result").innerHTML = "File sending failed.";
}
});
},
onFailure : function(response) {
that.controller.get("result").innerHTML = "FAILURE:"
+ response.reply;
}
});
}
FirstAssistant.prototype.setup = function() {
var libraries = MojoLoader.require({
name : "mediacapture",
version : "1.0"
});
this.mediaCaptureObj = libraries.mediacapture.MediaCapture();
this.controller.setupWidget("Select", {}, {
"label" : "Select",
"buttonClass" : "",
"disabled" : false
});
Mojo.Event.listen(this.controller.get("Select"), Mojo.Event.tap,
this.handleSelect.bind(this));
this.controller.setupWidget("Send", {}, {
"label" : "Send",
"buttonClass" : "affirmative",
"disabled" : false
});
Mojo.Event.listen(this.controller.get("Send"), Mojo.Event.tap,
this.handleSend.bind(this));
this.controller.setupWidget("File", {}, {
"disabled" : true
});
this.controller.setupWidget("ImageId");
this.controller.setupWidget("drawerId",
this.attributes = {
modelProperty: 'open',
unstyled: true
},
this.model = {open: false}
);
}
function FirstAssistant() {
var libraries = MojoLoader.require({
name : "mediacapture",
version : "1.0"
});
this.mediaCaptureObj = libraries.mediacapture.MediaCapture();
}
FirstAssistant.prototype.activate = function(event) {
this.controller.get('ImageId').mojo.manualSize(100,100);
};
FirstAssistant.prototype.deactivate = function(event) {
};
FirstAssistant.prototype.cleanup = function(event) {
};
</pre>The first function opens the file picker that allows you to select a file (an image in our case) from the file system. This filesystem to which the picker default goes is mounted under '/media'. This is also the place where normally all the user data is located. I was in luck that this file picker automatically shows a button to take a new picture, so with this I had everything in one. The result (see <a target="_blank" href="https://developer.palm.com/content/api/reference/mojo/classes/mojo-filepicker.html">here</a>) in case of an image contains the 'fullPath' to the image. Next, I set the path in the textfield, and set it on an image viewer which will show the image. Finally I open the 'drawer' so that everything becomes visible. <br />
<br />
The second method uses JQuery to do a POST Ajax call to the RESTFull webservice. It first calls the service to load our image and transforms it to base64, the rest is normal JQuery usage. Some notes: <br />
<ul><li>The the webOS application javascript imports (mojo/mojo-loader) have prototype (the JS framwork) build in. I don't know which version or how exactly, but you can easily do Ajax using prototype syntax. Since I'm more familiar with JQuery and I wanted to try if that also worked together, I did it this way (so I explicity imported JQuery in my index.html)</li>
<li>The file loaded is put into memory. Since its base64 it will probably be around 30% bigger then the initial file. I have no idea how much a JS in webOS (webkit) can handle by default, but I doubt I'll be able to send 100mb files this way.</li>
</ul><br />
You can download the Pre2 app together with the Java backend here: <a target="_blank" href="https://sourceforge.net/projects/webosphotouploa/files/webos-upload-photo.tar.gz/download">DOWNLOAD</a><br />
<br />
Btw; if you want to try this app (or any other app that requires your computer as server) on your phone, and you don't have any wireless connection, you can use the USB cable. See <a target="_blank" href="http://www.webos-internals.org/wiki/USBnet_Setup">here</a>.<br />
On ubuntu this works out of the box. You just type 'usbnet enable' on the phone, restart it, and ubuntu will automatically add a new NIC (called usb0). You can then reach the host computer from your pre2. If you want to reach the entire network, you'll have to iptables something like this:<br />
<br />
<pre class="brush: shell;">iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
iptables --append FORWARD --in-interface usb0 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward
</pre><br />
To conclude:<br />
<br />
Developing for the Pre2 is really fun. There is a lot of information to be found on the web, and the integration widgets/services are really, really good. As demonstrated, picking an image, taking a new photo, its all there out of the box. Getting the current GPS location, thats also 2 lines of JS etc... You can access almost every part on the Pre2 with some lines of Javascript. The fact that you write something like HTML and Javascript makes that its easy to develop and very lightweight. The SDK is also very good, works out of the box on linux, and the emulator makes it easy to develop without having to use a real phone.<br />
<br />
The dislikes: the eclipse plugin is very limited and did not work very stable. Every change you make requires to repackage the app and install it on the emulator or phone. This only takes some seconds, but thats still too much. The debugging capabilities are poor. I was hoping I could put a breakpoint in the JS file in eclipse, but that was not possible. You can put a breakpoint using the CLI and the palm-debug command, but thats not really appealing (and its very time consuming).<br />
<br />
Some images :<br />
<br />
<table><tr> <td colspan="2">Start the app. The right icon is the app in which the service lives. If you launch it you'll just get the empty screen of the application in which the service resides</td> </tr>
<tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-sTKWWsNu5GM/ThxhxcFOUiI/AAAAAAAAAK4/yrNhTNTlKCA/s1600/pre2-1.JPG" imageanchor="1" style=""><img border="0" height="400" width="266" src="http://2.bp.blogspot.com/-sTKWWsNu5GM/ThxhxcFOUiI/AAAAAAAAAK4/yrNhTNTlKCA/s400/pre2-1.JPG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-Sum7edrQzMM/Thxh5wNRX3I/AAAAAAAAALA/H7yVMjJ3lTw/s1600/pre2-2.JPG" imageanchor="1" style=""><img border="0" height="400" width="268" src="http://3.bp.blogspot.com/-Sum7edrQzMM/Thxh5wNRX3I/AAAAAAAAALA/H7yVMjJ3lTw/s400/pre2-2.JPG" /></a></div></td> </tr>
<tr> <td colspan="2">After pushing the 'select' button, the file picker opens. If running on a phone you can either select an existing photo or take a new one. We choose an existing (wallpaper):</td> </tr>
<tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-CJkqUSCuheA/ThxiOGdYU0I/AAAAAAAAALI/L_cBqtOctsg/s1600/pre2-3.JPG" imageanchor="1" style=""><img border="0" height="400" width="268" src="http://3.bp.blogspot.com/-CJkqUSCuheA/ThxiOGdYU0I/AAAAAAAAALI/L_cBqtOctsg/s400/pre2-3.JPG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-fyGKJF0h2Dw/ThxiRVvbjiI/AAAAAAAAALQ/HkvgPuQyMu0/s1600/pre2-4.JPG" imageanchor="1" style=""><img border="0" height="400" width="269" src="http://3.bp.blogspot.com/-fyGKJF0h2Dw/ThxiRVvbjiI/AAAAAAAAALQ/HkvgPuQyMu0/s400/pre2-4.JPG" /></a></div></td> </tr>
<tr> <td colspan="2">After selecting the picture it is shown on the screen. Select it by tapping 'open photo'. After that the image path is passed to our JS. It is then displayed in the image viewer scaled down to a kind of thumbnail:</td> </tr>
<tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-7N_YqDIOvsc/ThxiU3ibhKI/AAAAAAAAALY/9Rt2Nm3yu68/s1600/pre2-5.JPG" imageanchor="1" style=""><img border="0" height="400" width="266" src="http://1.bp.blogspot.com/-7N_YqDIOvsc/ThxiU3ibhKI/AAAAAAAAALY/9Rt2Nm3yu68/s400/pre2-5.JPG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-QjKUcSNr_wQ/ThxiWzFjRFI/AAAAAAAAALg/XVM1cofsxpA/s1600/pre2-6.JPG" imageanchor="1" style=""><img border="0" height="400" width="267" src="http://1.bp.blogspot.com/-QjKUcSNr_wQ/ThxiWzFjRFI/AAAAAAAAALg/XVM1cofsxpA/s400/pre2-6.JPG" /></a></div></td> </tr>
<tr> <td colspan="2">Tapping 'send' will submit it to our RESTful webservice. The status is shown in text at the bottom of the screen. After some seconds the browser automatically shows the uploaded image:</td> </tr>
<tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-l-j3wJrsaLM/ThxiZKcJf7I/AAAAAAAAALo/8_srpcd7AV8/s1600/pre2-7.JPG" imageanchor="1" style=""><img border="0" height="400" width="268" src="http://4.bp.blogspot.com/-l-j3wJrsaLM/ThxiZKcJf7I/AAAAAAAAALo/8_srpcd7AV8/s400/pre2-7.JPG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-PDYpH9Y_CuI/ThxibkT3cvI/AAAAAAAAALw/PJNHDQqJ-l8/s1600/pre2-8.JPG" imageanchor="1" style=""><img border="0" height="291" width="400" src="http://4.bp.blogspot.com/-PDYpH9Y_CuI/ThxibkT3cvI/AAAAAAAAALw/PJNHDQqJ-l8/s400/pre2-8.JPG" /></a></div></td> </tr>
</table>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-70474390468565911662011-05-28T16:24:00.000+02:002011-05-28T16:24:52.443+02:00Servlet 3.0: ARP, to push or not?With the Servlet 3.0 API out now for I while I wanted to check out its ARP (Asynchronous Request Processing) functionality. Briefly: ARP enables us to execute time consuming processing, while not holding on to our Servlet thread. So, I started doing some browsing and after a short while I ended up with more questions then I started with.<br />
<br />
This was thanks to a lot of self explanatory words such as: comet, cometd, tektite, meteor, asteroid, cavebear, grizzly, websocket, bayeux, ... ?!?! Btw; can you spot the two words not belonging in the row? To focus back on the Servlet 3.0 ARP functionality, I learned a couple of things: first, it is NOT a solution for server side push ~ Comet (all the other terms ARE related to it). For me this took a while to understand.<br />
<br />
If you look up any of the words I mention in combination with ARP, you will most definitely find a how-to that brings ARP in relation with server side push or at least compare them. So that made me confused, thinking that ARP was an implementation for it. Agreed, building a Comet implementation will benefit from ARP (Grizzly Comet API for example). But out of the box ARP is just not Comet and it should IMHO not be used in the same sentence.<br />
<br />
The Servlet 3.0 ARP functionality is limited to, well, asynchronous processing on the server side.If you look in the specification, they do not mention anything about underlying connection mechanisms. Terms as 'long polling' or 'streaming' (which are vital for a Comet implementation) are not mentioned in the specification.<br />
<br />
ARP is just a standardized mechanism for offloading your request processing in a different thread then the 'Servlet' thread. When the Servlet thread terminated, the connection is retained somewhere and it gives you a standard mechanism for getting back to the initial Servlet response from the 'asynchronous' thread to write (extra or not) data to the client.<br />
<br />
In fact, the client is even not aware that ARP is being used.<br />
From the clients point of view the connection stays open and it takes a long time to get the response (if you have a long running processing, of course). No changes on the client are required when you use ARP. When you use a real comet implementation however, changes are required one way or the other.<br />
<br />
Second, I needed to cut down on my enthusiasm about what it does do.<br />
This is what the specification says about it:<br />
<br />
<i style="font-size: smaller;">"Some times a filter and/or servlet is unable to complete the processing of a request<br />
without waiting for a resource or event before generating a response. For example, a servlet<br />
may need to wait for an available JDBC connection, for a response from a remote web service,<br />
for a JMS message, or for an application event, before proceeding to generate a response.<br />
Waiting within the servlet is an inefficient operation as it is a blocking operation that <br />
consumes a thread and other limited resources. Frequently a slow resource such as a database <br />
may have many threads blocked waiting for access and can cause thread starvation and poor <br />
quality of service for an entire web container.<br />
<br />
Servlet 3.0 introduces the ability for asynchronous processing of requests so that the<br />
thread may return to the container and perform other tasks. When asynchronous processing <br />
begins on the request, another thread or callback may either generate the response and call<br />
complete or dispatch the request so that it may run in the context of the container using the AsyncContext.dispatch method"</i><br />
<br />
Mmkay, first this seems weird. We start another thread to do our asynchronous processing, doing so we can release the Servlet thread. <br />
Isn't that like <insert funny zero operation analogy here> ?<br />
<br />
On resource usage level it is exactly the same. Using ARP you have no extra means in saving resources, like the difference between blocking and non blocking IO containers.<br />
The advantage using ARP is that when using a thread pool for its Servlets, it might give another request a chance by freeing up a thread from that pool.<br />
You moved the long running process to a thread from another thread pool. If this thread pool is full, it will not have any effect on (blocking) new requests.<br />
<br />
To illustrate this: suppose you have a Servlet thread pool of 100. You fire 200 requests at it at the same time.<br />
The first 100 requests are destined for LongRunningServlet and take a long time to process. The other 100 requests are for DoAlmostNohtingServlet, which processes very fast.<br />
Since our Servlet pool is already full with the first 100 (slow running) requests,the other (fast running requests) are queue'd and need to wait until a thread is released to the pool.<br />
<br />
If we would be adding ARP with an additional thread pool (lets call this one the ARP pool) of 10 threads, we would have this scenario:<br />
<br />
A request ending up in LongRunningServlet publishes an additional processing request to the ARP pool and terminates. The Servlet thread is returned to the Servlet pool almost directly. The result is that our 100 requests for LongRunningServlet are "processed" within seconds, freeing up threads for processing the request for DoAlmostNohtingServlet. The requests for DoAlmostNohtingServlet are then also processed withing some seconds.<br />
<br />
<ul><li>The requests DoAlmostNohtingServlet are processed in seconds, they (almost) did not have to wait for a Servlet thread </li>
<li>After a short time the Servlet thread pool is completely free again, open for serving new requests</li>
<li>In the mean time there is still a waiting queue for 90 requests for the ARP pool (10 requests are assigned to threads and are already processing).</li>
</ul><br />
So they will gradually get processed, but while they are processed the servlet thread pool is again completely free for new requests. To make some things clear:<br />
<br />
The connections to the clients of LongRunningServlet are still maintained. So the client is waiting untill its request for LongRunningServlet is processed by the ARP pool.<br />
Holding the connection (while the thread is busy in the ARP pool) does not consume a thread by itself.<br />
ARP will behave exactly the same in a blocking IO container as in a non blocking IO container:<br />
For a container using blocking IO, the 'Servlet thread' is the thread that got the Socket connection.<br />
<br />
A basic blocking server would do something like this:<br />
<br />
<pre class="brush: java;">while (true) {
final Socket socket = serverSocket.accept();
//Handof the socket to a 'servlet thread'
threadPool.submit(new Runnable() {
@Override
public void run() {
SomeHttpRequestObject httpRequestObject = parseRequest(socket.getInputStream());
process(httpRequestObject);
}
});
}
</pre><br />
The moral for a blocking IO Servlet container is that the socket (connection) is read in the Servlet thread.<br />
This means that a slow connection consumes a thread without anything happening.<br />
Suppose you have 1000 slow clients, none of these clients delivered a complete request yet, and at a given point 0 of those clients are sending data: you'll still take up 1000 threads for doing virtually nothing (in the assumption our Servlet thread pool is => 1000).<br />
<br />
With a non block IO server, connection reading takes only resources when there is actually something to read. This reading might happen in the Servlet thread as well, but as long as a request is not completely read and there is nothing more to read at that time, the thread is returned.<br />
<br />
So basically a servlet thread is only going to work (for a longer time) when a complete request has been read. Suppose you have 1000 slow clients, none of these clients delivered a complete request yet, and at a given point 0 of those clients are sending data: you'll use (virtually) 0 threads at time.<br />
<br />
Now, as you see the blocking/non blocking scenario plays before the request is handed over to the acutal Servlet. So using ARP is not directly influenced by it.<br />
<br />
To conclude: ARP allows you to offload a long running process via a separate thread pool. This releases the Servlet thread back to the pool allowing it to service new requests. By doing this your long running processes are not hogging the Servlet thread pool. The connection from the client that triggered the long running process is kept alive and the ARP API gives you the means to get back to that connection to write (additional) data to it when the long running process completed. ARP does not directly saves you any resources, but being able to offload request to a separate thread pool does give you more control about the number of threads maximum in use at a given point, without having other clients to wait to get served.<br />
<br />
Also, ARP serves as a scaleable base for possible Comet implementations , since a long polling request (to name one of the Comet connection strategies) does not hog a thread from the Servlet pool while waiting.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com5tag:blogger.com,1999:blog-3406746278542965235.post-6112541887839338512011-03-29T07:38:00.000+02:002011-03-29T08:17:50.111+02:00Personal serverFor personal use I need a small sized server that is reachable over the Internet. I use it to manage my personal emails, queue long downloads, serve access services (ftp, http, ...).<br />
<br />
My requirements:<br />
<ul><li>Reachability: it should be reachable from almost every location, even behind proxies and firewalls</li>
<li>Stability: it should be as stable as possible</li>
<li>Usability: it should be able to run any software that I like, without restrictions</li>
<li>Security: pretty clear I guess</li>
<li>Manageability: it should be manageable via different interfaces, offer possibilities to easy backup data etc<br />
<li>Pay-ability: at the lowest cost possible</li><br />
</ul>Bandwidth is not a tight requirement, since it will only be serving, well just me. I might put some pages on the web server, but even then images of heavy content will be placed somewhere on the Internet. Also, high availability is also not important, since it will only be for me (as long as there is enough stability). In the end I decided to install a server at home. Doing this I can also use it as a NAS to serve content, store backups and the likes via the home network. My standard ISP contract does not include fixed IP addresses. But, they do allow each port to be accessible from the Internet, so thats good (there are ISPs that block ports < 1024 without having an option to open them). The bandwidth is not very high, but that is ok for me. In physical measured speeds, it is about 7Mbps down and 418 Kbps up.
So with that in mind, I arranged the following:
<ol>
<li>Registered a .be TLD by a provider that has domain management</li>
<li>Put an old laptop, aka 'server', somewhere on my desk (who needs an UPS, if you have a laptop)</li>
<li>Register for a free domain alias service</li>
<li>Configure laptop with OS and other required software</li>
<li>Configure home router to make server accessible over the Internet</li>
</ol>The only extra cost for me is about 15euro/year for the TLD (domain+management), that's it. <p/>Ok, so how did I wire this up: <p/>First, the domain is a normal '.be' domain registered via a provider. The provider (aka registrar) registers the domain with the instance controlling the .be TLD zone's. They supply their own name servers as NS record, so my domain will be resolved by my provider name servers. Next, the provider gives me access to control my domain controlled by their name servers. This access is pretty elaborate, its a web interface right on top of the <a href="http://www.isc.org/software/bind" target="_blank">named</a> zone configuration. So I have full control and can actually configure it as it was running on my own name server, sweet. Since I do not have a fixed IP address, I'm not able to point a host name in my domain directly to my home server. I use a free dns alias service (<a href="http://www.dyndns.com/" target="_blank">for example</a>) that is able to update a host name entry directly when my IP address changes. Before declaring me foobar, I'll give some clarifications at this point: <ul><li>The dnsalias is also a name provider, just as my .be domainname provider. For free, they only give you a host within their own domain (like x.dnsalias.com, y.dynip.org, ...) and limited manageability. If you pay them, you can have both though, they register your domain and you can use their dns services to update it as you want. However, they do not support .be domains, since they are not an official .be registrar.</li>
<li>I could have dropped the '.be' domain and directly used the dnsalias. Now I have two name resolv's, a host on my TLD resolves to the dnsalias which then resolves to my home IP address. By directly using the dnsalias I save one resolve. But, I like to have my own .be domain. Furthermore, the dnsalias name is not as officially mine as a .be domain is. Even not when I pay for their services. (Now I can always decide to get a fixed IP address and point directly to that from my .be domain)</li>
<li>I could also have dropped the dnsalias and update my IP directly with my domain service provider. But they don't offer an interface to do that on an automated fashion.</li>
</ul>It took me 5 minutes to go through the free registration for a dnsalias, thats it. The fact that there are two hosts to be resolved are certainly not noticeable for normal usage. To map a host in my domain to the dnsalias, I cannot use a normal DNS 'A' record mapping that maps a host name to an ip address (or the other way around for reverse zones), since it is not possible to use a host name instead of an IP address in an A record. So luckily there exists something as a 'CNAME' record, which allows a host name in your domain to be linked with another host name. So on my domain provider I have something like this setup in my zone; <pre class="brush: shell;">host.domain.be. IN CNAME host.dnsalias.com
</pre>You can also use wildcards, so '*.domain.be. IN CNAME host.dnsalias.com' would resolve everything in *.domain.be to host.dnsalias.com.The dnsalias service has a normal A record mapping to my home ip address wit a very low TTL. If I do a zone scan, it looks like this: <pre class="brush: shell;">host.dnsalias.com. 60 IN A w.x.y.z
</pre>This means that if the information is propagated, it will be only be cached for 60 seconds on the intermediaries. Not what you typically want for a busy site. My server will run a small client that updates my IP address with the dns alias service each time it changes. It does that by using an external IP check service, that returns the Internet visible ip address. When that is changed over the last time, it sends out an update to the dnsalias service (+ my account information) with the new IP address. In the above zone snippet, the address w.x.y.z will then be updated.<p/>As operating system I choose Ubuntu (10.04.2 LTS 32bit). Its rock solid, supports all hardware on the server, has a great user support base and its pretty secure out of the box. However, I also like windows for its office and outlook for my emails. And some other windows only programs that just work better under windows. The laptop was originally installed with windows, so it had a windows XP cdkey. Also, windows XP also has nice native remote desktop support. The RDP protocol shares the clipboard, sounds, it reverse maps your hard drive over the same protocol (so on the VM you see a share of the client's hard drive from which you are connecting) and so on. Some will argue that a remote X server is maybe better, might be true, but an RDP client is available on any windows client. On ubuntu its also available by default. On other linux dists is probably a simple download. Setting up a remote X on a windows client will be more work/require more privileges I think. To resume, these are the steps I did to configure my ubuntu: <ol><li>Install <a href="#vmwareserver">VMware server</a>, v2.0.2-203138, used bridge Ethernet connection for VM</li>
<li>Install windows XP on VM, necessary software, and enabled remote desktop</li>
<li>Install <a href="#vmwareserver">ddclient</a> for automatic dns update</li>
<li>Configured my physical <a href="#wireless">wiress ethernet connection to autostart without logging in</a></li>
<li>Install <a href="#sshd">sshd</a>. Adjusted sshd config file to listen on two ports (22 and 443) and allow it to foward (more on that later)</li>
</ol><a name="vmwareserver">VMware server:</a> On previous installs of ubuntu (and VMWare server) I never had problems. However this time the installation failed. Thanks to the community I was able to pickup a patch for that: <a href="http://radu.cotescu.com/how-to-install-vmware-server-ubuntu-fedora-opensuse/" target="_blank">radu.cotescu.com</a>. After that VMware server installed without any problems.<p/><a name="vmwareserver">DDClient:</a> <pre class="brush: shell;">sudo apt-get install ddclient
sudo nano /etc/ddclient.conf
# Configuration file for ddclient generated by debconf
#
# /etc/ddclient.conf
protocol=dyndns2
use=web, web=checkip.dyndns.com, web-skip='IP Address'
server=members.dyndns.org
login=<your login>
password='<your password>'
yourhost.dnsalias.com
</pre>I followed this guide: <a href="https://help.ubuntu.com/community/DynamicDNS" target="_blank">ddns ubuntu</a><p/><a name="wireless">Wireless Ethernet connection to autostart:</a>The network manager is only started once you logon in X. So I needed something to connect the server to the wireless network the moment ubuntu was booted. I followed this <a href="http://ubuntuforums.org/showthread.php?t=318539" target="_blank">guide</a>. Basically it came to: <pre class="brush: shell;">sudo gedit /etc/network/interfaces
auto lo
iface lo inet loopback
auto wlan0
iface wlan0 inet static
address 192.168.0.2
gateway 192.168.0.1
dns-nameservers 195.238.2.22, 195.238.2.21
netmask 255.255.255.0
wpa-driver wext
wpa-conf managed
wpa-ssid Gateway
wpa-ap-scan 2
wpa-proto RSN
wpa-pairwise CCMP
wpa-group TKIP
wpa-key-mgmt WPA-PSK
wpa-psk <the key>
</pre>The wpa-psk key is generated by this command: <pre class="brush: shell;">wpa_passphrase <your_essid> <your_ascii_key>
</pre>With the command: <pre class="brush: shell;">iwlist scan
</pre>You should be able to find out the information you need from the AP you want to connect to. After that a network restart enabled my wireless on startup.<p/><a name="sshd">SSHD:</a> <pre class="brush: shell;">sudo apt-get install sshd
vi /etc/ssh/sshd_config
</pre>And add this: <pre class="brush: shell;"># What ports, IPs and protocols we listen for
Port 22
Port 443
GatewayPorts clientspecified
</pre>The last option will allow the client to specify target ports to which ssh should forward packets to (by default the sshd can only forward to the host it is running on. To forward to other targets, you need the 'GatewayPorts' as mentioned above).<p/>Ok! The only thing left was the accessibility. I could port map the RDP port from the XP VM directly to the Internet via the router. I could say that 3389 should be port forwarded to 192.168.0.3. However, I only wanted to open one port (besides HTTP) to the Internet, and preferably I want to shield the windows VM completely from the Internet. Also, as I want to access my services from everywhere, sometimes places just give you Internet access via an http proxy. From those places I would not be able to connect directly to the RDP service. To solve this, I tunnel everything I need over SSH. My server exposes only 3 ports: <ul><li>22 (SSH)</li>
<li>80 (http)</li>
<li>443 (SSH)</li>
</ul>The router is configured to port forward TCP 22,80 and 443 to the host OS on 192.168.0.2. Instead of running an HTTP SSL acceptor on 443 I configured SSHD to listen on two ports simultaneously; 443 and 22, so its not a typo. When I'm connecting from a remote location (me, being the client) I just need putty. Putty can be configured to connect directly (using port 22) and make a forward tunnel. Doing this I can choose a local port on the client that maps to a port on the target. Even better, I can map it to a port on any target, local network, or even back to the Internet. So, to connect to RDP, I need a tunnel mapping from <pre class="brush: shell;"><any local port> : 192.168.0.3 : 3389
</pre>Remember: the Internet does not know (route) 192.168.0.3. But this address is valid in the 'tunnel' that putty sets up. Putty first establishes a connection to ssh.mydomain.be and over that connection it is making a connection to 192.168.0.3 , so requests to 192.168.0.3 are send to the sshd, which then delivers them to the local network.
The total pictures looks like: <ul><li>I let putty connect to ssh.mydomain.be at port 22 (or 443, see below)</li>
<li>Putty creates a forward tunnel from localhost:xxxx (over 192.168.0.2) to 192.168.0.3:3389 via the tunnel (xxxx is 4000 in the screenshot below).</li>
</ul>In putty that looks like this: <table><tr> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-dtFXR9lmTc8/TZFu1aRjRyI/AAAAAAAAAIU/DsHCc63jY-M/s1600/putty1.PNG" imageanchor="1" style=""><img border="0" height="334" width="350" src="http://4.bp.blogspot.com/-dtFXR9lmTc8/TZFu1aRjRyI/AAAAAAAAAIU/DsHCc63jY-M/s400/putty1.PNG" /></a></div></td> <td><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-aIl6eF1EgEw/TZFu-bSPrHI/AAAAAAAAAIc/hyZWyN5r76s/s1600/putty2.png" imageanchor="1" style=""><img border="0" height="334" width="350" src="http://4.bp.blogspot.com/-aIl6eF1EgEw/TZFu-bSPrHI/AAAAAAAAAIc/hyZWyN5r76s/s400/putty2.png" /></a></div><p/></td> </tr>
</table>On the client computer from which I'm connecting from, I point my RDP client to localhost:4000 and the connection is established. The nice thing about all of this, is that there is an option in putty to tunnel my tunnel over an http proxy. So if I'm not allowed to go out on 22 directly, I configure putty to talk to the proxy to send out my packets. If you enable HTTP proxying on putty, putty sends an HTTP CONNECT <targethost:port> to the proxy. The proxy will then tunnel your request further to the target.<p/>However, sometimes proxies disallow to tunnel to a target port as '22(ssh)'. An easy trick is spawning the sshd on a second, SSL/TLS port 443. The HTTP handshake for a SSL/TLS connection is the same as it is for a proxied SSH request (its also tunneled by HTTP CONNECT). So actually the proxy thinks you are going to SSL (because of the port), but you are not. To do this, just let putty connect to port 443 instead of 22 (as shown in the first image above). Next you have to tell putty that it should use a proxy: <div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-9nOfpc2wSxI/TZFvCAeyz7I/AAAAAAAAAIk/VrCGNBIqz5k/s1600/putty3.PNG" imageanchor="1" style=""><img border="0" height="334" width="350" src="http://2.bp.blogspot.com/-9nOfpc2wSxI/TZFvCAeyz7I/AAAAAAAAAIk/VrCGNBIqz5k/s400/putty3.PNG" /></a></div><p/>Before declaring me foobar (again), I'll give some clarifications at this point: <ul><li>What I'm doing here is building a "poor man's" VPN to a certain extend. However, VPNs work with different protocols, requires a VPN client (and a VPN server). It also requires the network infrastructure to 'allow' to setup a VPN. So suppose you want to connect your (own) office with your home network, a real VPN would definitely be better and more scalable. However, this setup needs to work as lightweight as possible on any type of client (possibly not managed by me) and preferably on any type of network.</li>
<li>A really tightend up proxy will disover fast (even if you do it over 443) that you are not SSL/TLS-ing. In that case I could still add an extra module which will tunnel my ssh tunnel in an ssl tunnel tunneled over the proxy. By doing that the proxy is not able to distinguish your session from a SSL/TLS that for example has been started by a clients browser. Since I did not yet meet such tightned proxies, and this setup would require some additional client software as well, I will leave it like this until its really necessary some day.</li>
</ul>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-86835651121769529602011-03-26T03:21:00.000+01:002011-03-26T16:38:24.708+01:00java.text.CollatorSome time ago I got introduced to a part of the Java text API that was unexplored territory for me: the <a target="_blank" href="http://download.oracle.com/javase/6/docs/api/java/text/Collator.html">Collator</a>.<br />
<br />
Languages imply more complexity then one on first sight would think (check <a target="_blank" href="http://www.unicode.org/versions/Unicode6.0.0/UnicodeStandard-6.0.pdf">this</a> if you have any doubt).<br />
The main usage of the Collator is to help us with a part of that linguistic complexity, more specifically locale sensitive comparison. It implements the collation specification defined by the <a target="_blank" href="http://unicode.org/reports/tr10/">Unicode</a><br />
<br />
The Java Collator roughly does these things:<br />
<br />
<ul><li>Canonicalization of canonical equivalent characters<br />
</li>
<li>Multi level comparison</li>
</ul><br />
Comparing Java based Strings works by comparing their Unicode code point that maps with the character. This would mean that the position of the character in the Unicode code charts specifies the sorting weight, but that is not the case. Languages might have different sorting weights for the exact same characters.<br />
<br />
For example, if you don't know anything about the German language, you might expect that ß (\u00DF) is sorted as it was a 'b' or 'B'. That is not correct, since its actually represents the combination 'ss'. But even knowing this, a standard comparison with ß would yield false results since its code point is higher then a normal 's'. So in the end it will not be sorted as an 'ss' but it will be sorted as it was 'higher' then 'z'.<br />
<br />
Multi level comparison solves this by offering 4 comparison levels: base letters, accents, case, punctuations. If the first level is used, only base character differences are considered. With the second level base characters as well as accents are considered significant, etc<br />
<br />
Note: the Collator apparently does not support punctuation.<br />
<br />
<pre class="brush: java;">System.out.println("a equals b -> " + (collator.compare("a", "b")==0 ? "true":"false"));
System.out.println("a equals à -> " + (collator.compare("a", "à")==0 ? "true":"false"));
System.out.println("A equals a -> " + (collator.compare("a", "A")==0 ? "true":"false"));
</pre><br />
With collator.setStrength(Collator.PRIMARY):<br />
<pre class="brush: java;">a equals b -> false
a equals à -> true
A equals a -> true</pre><br />
With collator.setStrength(Collator.SECONDARY);<br />
<pre class="brush: java;">a equals b -> false
a equals à -> false
A equals a -> true</pre><br />
With collator.setStrength(Collator.TERTIARY);<br />
<pre class="brush: java;">a equals b -> false
a equals à -> false
A equals a -> false</pre><br />
Our first use case for which we used the Collator was for the first function; canonicalization. Unicode foresees different ways of representing certain characters. For example; ü is identified by a single code point \u00FC and thus a single character. However, it is also possible to form ü with a character + diacritical mark(<a href="#note1">°</a>): u (\u0075) and ¨ (\u0308).<br />
<br />
It makes sense if you think about it, on a classic typewriter you would also form ü by first printing u, go back one position and then print ¨. The ¨ is a so called invisible character on the type writer. On our keyboard we can do the same. You can press the button marked ü or you can: <altgr> + <¨> + <u> which gives you ü.<br />
<br />
The end result (on your screen at least) is the same disregarding how you form ü, however, it is stored differently. The single character ü would be saved as: 0x00FC. If you would have formed ü by typing ¨ followed by u, it would be saved as: 0x0075 0x0308<br />
<br />
As long as you just want print those characters, in a browser, console, text editor, ... you can (hopefully) rely on that software to display it right. However, if you are writing Java and want to do operations with character streams containing such characters it becomes tricky.<br />
<br />
Example (<a href="#note2">°°</a>): lets take a typical German word such as "abgaskrümmerdichtung":<br />
<pre class="brush: java;">String single = "abgaskr\u00FCmmerdichtung";
String combined = "abgaskr\u0075\u0308mmerdichtung";
System.out.println("Single equals combined? " + single.equals(combined));
System.out.println("Single: " + single);
System.out.println("Combined: " + combined);
</pre><br />
The first line will say that they are not equal: Single equals combined? false<br />
However, when both are displayed, they look exactly the same:<br />
<br />
Single: abgaskrümmerdichtung<br />
Combined: abgaskrümmerdichtung<br />
<br />
Our software needs comparison on a higher level rather then pure code point comparison. It needs to be canonicalized first, by something that knows that \u0075\u0308 is in fact \u00FC. Collator to the rescue:<br />
<br />
<pre class="brush: java;">collator.setDecomposition(Collator.CANONICAL_DECOMPOSITION);
String single = "abgaskr\u00FCmmerdichtung";
String combined = "abgaskr\u0075\u0308mmerdichtung";
System.out.println("Single equals combined? " + (collator.compare(single, combined) == 0 ? "true": "false"));
</pre><br />
This will print; Single equals combined? true<br />
<br />
A second use-case where the Collator came in handy: we were in need to map characters from ISO8859-1 to 7bit ASCII. ISO8859-1 contains several accented characters that do not exist in 7bit ASCII. Our goal is to map these characters to their canonical equivalent that is supported in 7bit ASCII. For example: "çéàëê" could be mapped to "ceaee". Of course, other characters for which no obvious equivalence exist cannot me mapped (and will be converted as '?')<br />
<br />
Remember: Java uses Unicode and UTF16 as encoding. ASCII and the ISO8859 family are both character maps and encodings in one. Unicode and ISO8859-1 share the same code points for the first 256 glyphs. They are also compatible on encoding level: if you take the letter 'A' and save it (codepoint + encoding = ISO8859) and you decode it as Unicode/UTF8, it will still print 'A'. (this is not the case with UTF16/32). ASCII only shares the first 128 glyphs (extended, 8bit, ASCII is not compatible with Unicode or ISO8859-1).<br />
<pre class="brush: java;">String ascii7Characters = " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~";
Collator ascii7Collator = Collator.getInstance(new Locale("nl", "BE"));
ascii7Collator.setStrength(Collator.PRIMARY);
Map<CollationKey, Character> ascii7CollationMappings = new HashMap<CollationKey, Character>();
for (char c : ascii7Characters.toCharArray()) {
ascii7CollationMappings.put(ascii7Collator.getCollationKey(String.valueOf(c)), c);
}
String accented = "çéàëê";
StringBuilder ascii7 = new StringBuilder();
for (Character character : accented.toCharArray()) {
Character canonicalizedCharacter = ascii7CollationMappings.get(ascii7Collator.getCollationKey(String.valueOf(character)));
ascii7.append(Character.isUpperCase(character) ? Character.toUpperCase(canonicalizedCharacter): canonicalizedCharacter);
}
System.out.println("ISO8859-1 converted to 7bit ASCII:" + ascii7.toString());
</pre>This will print: ISO8859-1 converted to 7bit ASCII:ceaee<br />
<br />
The String created on line 12 is of course Unicode (and encoded in UTF16, but not relevant now).<br />
But since these characters are in the range which is equal between ISO8859-1 and Unicode, it actually does not matter. In real life we would be reading in a byte stream which is explicitly decoded as ISO8859-1: <br />
<pre class="brush: java;">String accented = new String(inputInIso8859_1, "ISO8859-1");
</pre><br />
What we did here is use the collator its <a target="_blank" href="http://download.oracle.com/javase/6/docs/api/java/text/CollationKey.html">CollationKey</a> and bind it to our normalized 7bit ASCII character. These key are the canonicalized form of the character, depending on the strength and decomposition values you configured the Collator with. Characters that are canonical equal will also have the same CollationKey. You can use the CollactionKey for linguistically correct sorting/searching, since it will yield the correct order based upon the <a target="_blank" href="http://download.oracle.com/javase/6/docs/api/java/util/Locale.html">Locale</a> you initialized the Collator with (it implements Comparable).<br />
<br />
<span style="font-size:80%"><a name="note1">(°)</a>Diacritical marks are special glyphs, in that way that they are combined with other glyphs and say something about the intonation. For example: <`> can be considered a diacritical mark.<br />
</span><br />
<span style="font-size:80%"><a name="note2">(°°)</a>The so called code points named in this text refer to glyphs in the Unicode map. They are shown in Java escaped hexadecimal form notation, so \u + 16bit hex. The encoded form is UTF16, with the given examples this means that it is 16bit per character and the code point matches with the encoded form (since the code points in the examples are between \u0000...\uD7FF and \uE000...\uFFFF)</span>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0tag:blogger.com,1999:blog-3406746278542965235.post-89293286352801498932010-12-02T12:20:00.000+01:002011-02-08T13:38:07.892+01:00That pesky HibernateYesterday a collegue of mine asked if I could build some functionality to physically delete records in our database. <br />
Since deleting things is always a good idea, I immediately started to work on his request.<br />
As we are using an ORM (Hibernate) to manage our relational persistence, this would just be a matter of adding a single line of code:<br />
<br />
<pre class="brush: java;">session.delete(someEntity);
</pre><br />
And all the Hibernate magic combined with some cascading would take care of the deletion.<br />
So I concluded my assignment fast, and got the well know warm cosy feeling: 'job wel done'.<br />
However, suddenly I realised that the combinaton: "hibernate, magic, job well done, warm cosy feeling" has put me into problems before.<br />
In fact, I remeber this since last time I was in that position I ended up feeling exactly like <a target="_blank" href="http://www.theluxuryspot.com/wp-content/uploads/2010/03/Scarface-Al-Pacino_l.jpg">this</a> man did.<br />
<br />
Just to be sure I went back to the code and configured my logging to print out what Hibernate was doing.<br />
As it turned out, something bad was going on. It seems that also Hibernate has difficulties deleting relationships (thats a line to think about).<br />
<br />
The problem is that each child is deleted one by one.<br />
Lets say you have two entities: Person and Product. There is a one to many relationship between Person and Product.<br />
The relationship is uni-directional from Person to Product, mapped as a Set, managed by Person (as there is only one side) and the cascading is set to all-delete-orphan.<br />
<br />
<pre class="brush: xml;"><hibernate-mapping default-access="field" >
<class name="entities.Person" table="person">
<id name="id" column="id" access="property">
<generator class="native"/>
</id>
<set name="products" cascade="all-delete-orphan">
<key column="person_id" not-null="true" update="false" foreign-key="person_fk"/>
<one-to-many class="entities.Product"/>
</set>
</class>
</hibernate-mapping>
</pre><br />
<pre class="brush: xml;"><hibernate-mapping default-access="field" >
<class name="entities.Product" table="product">
<id name="id" column="id" access="property">
<generator class="native"/>
</id>
<property name="name" column="name"/>
</class>
</hibernate-mapping>
</pre><br />
If you delete a Person object which has 5 products, you will see that pesky Hibernate doing this:<br />
<br />
<pre>31398 [main] DEBUG org.hibernate.SQL - delete from product where id=?
31399 [main] DEBUG org.hibernate.SQL - delete from product where id=?
31399 [main] DEBUG org.hibernate.SQL - delete from product where id=?
31399 [main] DEBUG org.hibernate.SQL - delete from product where id=?
31399 [main] DEBUG org.hibernate.SQL - delete from product where id=?
31400 [main] DEBUG org.hibernate.SQL - delete from person where id=?
</pre><br />
In some conditions this might be acceptable, however, in our case there could be thousands of childs records.<br />
I do know that with bulk operations and large datasets Hibernate is maybe not the best choice.<br />
However, we configured Hibernate that it just worked for our case.<br />
<br />
For example; in our real scenario the "lazy" attribute on the Set has been set to "Extra"<br />
This allows to get a count on the child Collection without Hibernate loading every child.<br />
Other bulk operations we managed by doing HQL. So in the end it worked out very good (and fast) so we where happy with the ORM functionality without having to pay performance costs.<br />
So I was not planning to let this problem become a performance bottleneck.<br />
<br />
As it turns out this 'problem' is very known one and there is lots to be found on the internet. I learned that Hibernate should normally be able to do a 'single shot delete'. <br />
It is even mentioned in the manual! So I tried every possible combination, but could not force hibernate to do that 'single shot delete'.<br />
<br />
Secondly I tried to do it the 'bulk' way by just issuing some delete HQL.<br />
In our case its not a problem that it bypasses the Session, since the transaction consists only out of the delete.<br />
<br />
YAP! As it turned out, in my real scenario the Child has a Hibernate <map> relation to a properties table.<br />
When doing an HQL delete, the cascading on the <map> is not respected. This seems to be a issue as well (<a target="_blank" href="http://opensource.atlassian.com/projects/hibernate/browse/HHH-695">here</a>).<br />
I also tried to delete the <map> manually using HQL, but that doesn't work since there is no entity to 'grab' in the query.<br />
<br />
Thirdly, I learned about the "on-delete" attribute which you can set on the <key> element in the <Set>.<br />
The only requirement is that the Set must be the non-managed side of the relation (so inverse must be 'true').<br />
In my Person/Product example this is not the case (as there is only one side).<br />
But in my real scenario, which is slightly different, this is the case: the association is mapped bi-directionally and the Child is owner of the relationship.<br />
<br />
If you enable the on-delete = "cascade" Hibernate would leave the deletion to the database.<br />
In other words, it will depend on the database cascading delete, so your table DDL should have the 'on delete cascade' in its FK constraint DDL.<br />
Cool note, that if you use hbm2ddl Hibernate generates the correct DDL including the on delete cascade.<br />
<br />
Good, good! The only thing I now had to do was to tell our db administrator to add the 'on delete cascade'.<br />
<br />
But wait, I also still have the <map>, So I tried to add the 'on-delete= cascade' there as well.<br />
BUT! on-delete = 'cascade' can only be added on inverse associations!<br />
The <map> is mapped uni-directional and is therefore always the owner. So, bad luck once again, this did not work.<br />
<br />
Agreed: at this point I could refactor the <map> construct to a normal stand alone Entity and make a bidrectional one-to-many/many-to-one.<br />
But that would imply many hours refactoring for something that was just working fine, so I felt not doing this.<br />
<br />
Fourthly (and finally) I decided to just bypass Hibernate completely for this delete.<br />
I created a named query :<br />
<br />
<pre class="brush: xml;"><sql-query name="deletePerson">delete from person where id = :personId</sql-query>
</pre><br />
I asked the database administrators to add 'on delete cascade' to the FK's of Product and the map table.<br />
To be able to test this in our in-memory database, I added some extra DDL in the Person mapping:<br />
<br />
<pre class="brush: xml;"><database-object>
<create>
alter table product drop constraint person_fk
alter table product add constraint person_fk foreign key (person_id) references person(person_id) on delete cascade
<!-- same here for the <map> relation-->
</create>
<drop/>
</database-object>
</pre><br />
This allows to simulate the database cascading in my tests (which run against an in-memory database).<br />
So to conclude my journey, the steps you can try if you want a fast delete:<br />
<br />
1. Do not use Hibernate<br />
2. Just use the not working one-shot-delete, so you don't have to complain about Hibernate in some blog entry<br />
<br />
Now as for your real options:<br />
<br />
1. Map your 'one' side of the association with on-delete="cascade" and alter the FK constraints in the database with cascading<br />
2. Use HQL to delete the entities yourself (do not forget this is a bulk operation and the delete is not reflected in the Session!)<br />
3. Bypass Hibernate and use plain SQL + database cascading (same remark about the session)<br />
<br />
Oh, btw, if you use the first suggestion, be aware that (even if you are using lazy true/extra on the association) Hibernate always loads the complete association before issuing the delete. Ain't that cute or what?<br />
So if you have large datasets going on and you want performance, afaik you are limited to 2 and 3.Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com5tag:blogger.com,1999:blog-3406746278542965235.post-70494308471244677272010-11-20T03:40:00.000+01:002011-02-09T13:03:06.435+01:00ImmutabilityJava Puzzlers, the last presentation thursday at Devoxx, came with some new, or at least some refurbished, puzzlers to melt our minds. <br />
One however did melt something away in my mind. It had to do with object immutability.<br />
<br />
The puzzler goes like this:<br />
<br />
Supose you have map, lets say an EnumMap or ConcurrentHashMap.<br />
<br />
You put some values in that map, and next you obtain the entry set. <br />
Now, instead of just doing something with the entries, you add this set to another Collection of type Set using the addAll() method (or constructor, which in fact calls addAll for you).<br />
<br />
In code this might look like this:<br />
<br />
<pre class="brush: java;">Map<String, String> map = new ConcurrentHashMap<String, String>();
map.put("one", "two");
map.put("two", "one");
Set<Entry<String, String>> set = new HashSet<Entry<String, String>>(map.entrySet());
System.err.println(set.size());
</pre><br />
<span style="font-size: x-small;">Note: that the actual presented puzzler was in a more entertaining form</span><br />
<br />
You will notice that the set.size() will return 1, and not 2.<br />
The reason behind is: lack of clarity in the API and an attempted optimization by the implementation.<br />
<br />
The API does not say anything specific about the mutability of the Entry objects returned in the Set.<br />
Some Map implemenations made the choice of a 'singleton' Entry object instead of creating new immutable Entry objects for each value in the map.<br />
<br />
<br />
What actually happens is that they change the content of the one and only created Entry for each map value, by calling setters on that sole Entry object.<br />
<br />
So if a map has 1000 elements, only one Entry object will exist.<br />
When the map is dumped to a set of Entries, the map will have called 1000x setKey and setValue on that single Entry object for each element in the map.<br />
<br />
Funny side effect that if you change the values in the map to something like this:<br />
<br />
<pre class="brush: java;">map.put("keyOne", "valueOne");
map.put("keyTwo", "valueTwo");
</pre><br />
<div style="font-family: inherit;">The set.size() DOES print 2 :-)</div><div style="font-family: inherit;">What happens is that the map implementaton uses this construct to detect if an entry already exists:</div><div style="font-family: "Courier New",Courier,monospace;"><span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span></div><br />
<pre class="brush: java;">if (e.hash == hash && ((k = e.key) == key || key.equals(k)))
</pre><br />
<div style="font-family: inherit;">So, the hashCode must be the same AND they must be equal, either through reference or object equality.</div><div style="font-family: inherit;"><br />
</div><div style="font-family: inherit;">In the first case, the hashes were equal and the reference of key was also the same.</div><div style="font-family: inherit;">The hashCode on an Entry object is calculated by taking the hashCode of the key exorred with the hashCode of the value.</div><div style="font-family: inherit;">In the first case the hashCode of 'one' exorred with the hashCode of 'two' is the same as the hashCode of 'two' exorred with the hashCode of 'one'.</div><div style="font-family: inherit;"><br />
</div><div style="font-family: inherit;">Allthough they are distinct Entries (there will not be object equal) the hashCode is the same which is also allowed by the hashCode contract</div><div style="font-family: inherit;"></div><div style="font-family: inherit;"><span style="font-size: xx-small;">Remember that two objects that are equal must produce the same hashCode, but two non equal objects are not required to produce different hashCodes</span> </div><br />
<div style="font-family: inherit;"></div><div style="font-family: inherit;"></div><div style="font-family: inherit;">For the second case, the hash actually differs, so they don't bother looking further and the entry is considered as 'distinct'.</div><div style="font-family: inherit;">Of course if you would iterate the set, you would find that the key and value of the Entry object are twice the same.</div><div style="font-family: inherit;"><br />
</div>Its unlikley that you run into this problem in your code base, but if you do, it will guarantee long cosy hours of debug.<br />
<div style="font-family: inherit;">The reasons why I find this particular 'bug' so interesting are its diverse lessons that we can learn from it:</div><div style="font-family: inherit;"></div>1. Premature micro optimization comes at a cost. You could argue any optimization is good at this level . Maybe, but I know for sure that for us as application developers we should not attempt such optimizations of our own. You risk creating pitfalls beyond your knowledge while (als in most cases) the gains by such optimizations are very hard to prove. <br />
<br />
2. Immutability plays an important role in your code. It can avoid 'bugs' such as this one. It gives you a signature when handling an object that it is guaranteerd to be in the same state as it was created with.<br />
<br />
<div style="font-family: inherit;">3. Basic design and code rules apply for everyone. It also shows that it can be forgotten in any project. In this case both the API designer as well as the people who implemented it did not question this.</div>Koen Serneelshttp://www.blogger.com/profile/17324386688560910205noreply@blogger.com0