Turning ESP8266 WiFi off to save power (and turn it on later)

· Matti

There are lot of discussions how to turn ESP8266 WiFi off and on over there. Luckily, the solution is quite straightforward.

// Set WiFi credentials
#define WIFI_SSID "your-wifi-ssid"
#define WIFI_PASS "your-wifi-pass"

void disableWifi(){
    Serial.print(millis());
    Serial.print("\tdisableWifi\t");

    WiFi.disconnect();
    WiFi.mode(WIFI_OFF);
    WiFi.forceSleepBegin();
    yield();

    Serial.println("OK");
}

void enableWifi(){
    Serial.print(millis());
    Serial.print("\tenableWifi");

    WiFi.forceSleepWake();
    yield();
    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_SSID, WIFI_PASS);

    while (WiFi.status() != WL_CONNECTED) {
        digitalWrite(LED_BUILTIN, LOW);
        delay(250);
        digitalWrite(LED_BUILTIN, HIGH);
        delay(250);
        Serial.print(".");
    }

    Serial.println(WiFi.localIP());
}

The most useful and in-depth study has been carried out by OppoverBakke. In the first post he studies the power use of ESP8266 and in the second one he suggests the correct order of WiFi.forceSleepWake() and WiFi.mode() calls. The yield() call is required to let ESP ROM to get its internal things straight.

Further references:

Interfacing ESP8266 and Peacefair PZEM-017 with ModBUS RTU

· Matti

This brief note explains how to connect ESP8266 and Peacefair PZEM-017 to read voltage and current from the sensor over Modbus RTU (the RS485 based serial protocol).

Parts List

  • Wemos D1 (ESP 8266)
  • MAX3485
  • 120 ohm resistor (used 100 ohm for testing)

Connections

Code Example

#include          // https://github.com/emelianov/modbus-esp8266
#include 

// ESP8266 pins
#define D6 12
#define D7 13
#define D8 15

// PZEM-017 Shunt Range
#define PZEM_SHUNT_100A 0x0000
#define PZEM_SHUNT_50A  0x0001
#define PZEM_SHUNT_200A 0x0002
#define PZEM_SHUNT_300A 0x0003

// My settings
#define SERIAL_RX D8
#define SERIAL_TX D7
#define SERIAL_CONTROL D6
#define PZEM_MODBUS_ID 0x01
#define PZEM_SHUNT PZEM_SHUNT_200A

SoftwareSerial swSerial(SERIAL_RX, SERIAL_TX);

ModbusRTU modbus;
int modbusLastStatusCode = -1;

bool modbusStatusCallback(Modbus::ResultCode event, uint16_t transactionId, void* data) {
  // Serial.printf_P("Request result: 0x%02X, Mem: %d\n", event, ESP.getFreeHeap());

  modbusLastStatusCode = uint(event);

  return true;
}

void setup() {
  Serial.begin(115200);

  swSerial.begin(9600, SWSERIAL_8N1);
  modbus.begin(&swSerial, SERIAL_CONTROL);

  modbus.client();

  Serial.print("Setting shut value...");

  // The write resulted always 0xE4 (timeout), so no callback here
  modbus.writeHreg(PZEM_MODBUS_ID, 0x0003, PZEM_SHUNT);
  modbus.task();

  Serial.println("OK");

  Serial.println("Setup finished");
}

uint16_t registers[2];
float voltage;
float current;

void loop() {
  if (!modbus.slave()) {
    modbus.readIreg(PZEM_MODBUS_ID, 0, registers, 2, modbusStatusCallback);
    if (modbusLastStatusCode == 0) {
        voltage = registers[0] / 100.0;
        current = registers[1] / 100.0;
        Serial.print(voltage);
        Serial.print("V, ");
        Serial.print(current);
        Serial.println("A");
    }
    else {
        Serial.printf_P("Result code: 0x%02X\n", modbusLastStatusCode);
    }
  }
  modbus.task();
  yield();
}

For PlatformIO users, the platformio.ini:

[platformio]
default_envs =
    esp8266

[env]
lib_deps = 
    https://github.com/emelianov/modbus-esp8266
    
[env:esp8266]
platform = espressif8266
framework = arduino
board = d1
lib_ldf_mode = deep
monitor_speed = 115200
upload_speed = 460800

 

 

 

 

Finding and Building a SensESP/SignalK Sensor Hardware

· Matti

Building a microcontroller based sensor for my yacht appeared to be an easy task. During last year I have learned a lot and thought it would be nice to share some ideas.

From InfluxDB to SignalK

My initial choice for an architecture was InfluxDB. It appeared to offer everything you would want:

  • Installable as an Docker image
  • Lightweight enough to run on Raspberry Pi
  • A large variety of diagrams to choose from
  • Library to ESP8266 (Wemos D1 mini)

At this point the SignalK server and SensESP looked like too complicated and bloat for me. However, after following my tinkering for the first season my co-founder wanted something more professional-looking. He was also worried that a 100% DIY project would be a problem when we had to look for a new owner to our boat.

It took me a while to get myself familiar with the aforementioned framework. I even tried to write my own implementation of the SignalK client for ESP8266. Finally I gave up and did a showcase of some SensESP sensors producing random values and SignalK server with the default Instrument Panel. That did the trick.

Finding a Decent Case

Building the first sensors on the breadboard was an easy task but finding a decent case was harder than you could think. After playing with the first prototypes I came up with the following list of requirements:

  • The case should be waterproof(ish) and durable enough to survive the marine life. Our engine room is theoretically dry but there is always moisture on the boats.
  • No rusting metal parts. The screws (if any) should be plastic or stainless steel.
  • There should mounting lugs for wall mounting. It is not acceptable to mount the sensor by drilling the through its bottom as the circuit board must be removed to access the holes/screws.
  • There should be enough space for connectors.
  • No matter how nice over-the-air updates you might have at least I definitely have to have an easy access to the microcontroller. The housing should be easily opened.
  • Clear case would show the LED signals and help troubleshooting the sensors.
  • As we’re communicating on wifi it is best to avoid aluminium cases.

I tried to find a proper housing from the local electronics webshops and purchased some promising cases. Typically the screws securing the cover were in the bottom. In the prototype (well, it was supposed to be a production version but…) shown in the picture the I replaced the original screws with stainless steel threaded rods and nylock nuts. It worked all right but was a bit too laborious.

An early prototype with 4-channel ADS1115 voltage meter.

Finally I managed to find cases I had been looking for. I ordered one larger and one smaller box for and start. It turned out that they satisfied all my requirements.

Powering the Board

My first idea about powering the sensors was to build a old-school voltage regulator LM7805 in each of the devices (see this blog post for an example). With this I could use the yacht’s 24V DC and possibly avoid some wiring.

However, I soon found out that this cause me a lot of work and increased the risk for mistakes in soldering. I actually fried some capacitors when making this!

The next level in this path was to order an adjustable voltage regulator component from Asia. I bought a bag of these from Aliexpress but they turned out to be crap. Or maybe I could not install them, you never know.

These setbacks made me think of alternatives. I had noticed that still engineered a 24 V power cable to each of the sensors so I could power off the devices if required. My hope of somewhat utilising the existing power cables did not prove to work. Distributing the 24-to-5 DC-DC step-down made the sensors more compilated, large and prone to errors than installing the 5 V power network for the sensors.

 Finally I ordered a DIN rail installable DC-DC converter which gave me a stable five volts. It even has short circuit protection which has saved me a couple of times.

Connectors

In the very first experiments I drilled a hole to the housing and connected the wires to screw terminals on the board. While my goal was build the sensor at home and spend minimal time onboard installed the screw terminals outside the housing as can be seen in the early prototype above. This proved to be functional but required manual steps which I wanted to avoid. You have to drill holes for wires and screws securing the terminal bar and use a silicone to seal the holes. Especially the latter was time-consuming and messy.

I tried to find waterproof connectors which I could use but they turned out to be hard to find. As I wanted to housing cover to be as removable as possible without any wirings the connector should be small enough to be installable to the bottom part of the box. Again, if I had connectors to gauge and power I would need two connectors which would mean more soldering and drilling. I would have to solder tiny connectors onboard which I wanted to avoid as much as possible.

My current solution is to use stainless steel screws going through the housing. The wires both inside and outside have ring terminals. To secure the connection I solder all connectors to their wires.

While I don’t have long lasting experience of the solutions it appears to be quite nice at least to install. The Abiko connectors are widely available, robust and easily installable onboard. The screw terminals allow me to attach multiple wires to them. I can chain the power lines from sensor to another or use common ground or power for sensor and gauge (e.g. the Hall sensor for revolution measurement).

Building a Current Sensor

In this example I’m building a case for a current sensor. It has Wemos D1 mini with ADS1115 powered by SensESP firmware. The circuit board is made by Aisler. I have made sensors with prototyping board but again, there are a lot of soldering and tinkering which may break in the marine environment. While the Aisler prices are ridiculously low I wonder why bother.

The abovementioned cases look looks this. The white strip is for making the case waterproof. There is a groove on the cover for this.

All the wires connecting to the circuit board have ring terminals. The wires are soldered to the board. I avoid using screw terminals to avoid the risk of loosening connections.

The housing has seats for the screws. As I don’t need them I drill them away.

Drilling holes for the connector screws. I use 4 mm drill for M4 screws.

I use velcro strip to attach the circuit board. Again, I don’t have long-lasting experience on this but the installation is fast and so far the boards have remained in their places. Before attaching the strip I make sure the bottom of the circuit board is as flat as possible.

Attach the velcro strips.

Now it is time install the connecting screws. I use hex screws as they can be fastened with a proper tool in 90 degree angle. Regular philips or torx screws would be problematic to hold inside the case. So far I haven’t used any sealant for securing the holes. All the nuts are nylock to avoid loosening connections.

 

Normally I avoid putting any connectors to the cover. This makes it easy to remove the cover to grab the microcontroller. In this current sensor I decided to install the shunt to the cover to make it easier to make load connection.

The sensor has four connectors.

  • The screws on the shunt are for the load to be measured. If you’re not familiar with shunts take a look at this video.
  • The two screws on the bottom are for powering the board (5V DC).

You yourself know what the particular sensor is by heart but if you care the future boat owners or engineers it is good idea to document the sensor and connectors.

These two sensors are measuring the fuel and black water tanks. They use the ESP8266 internal ADC to measure the resistance of the gauges.

Wemos D1 and MAX6676 Thermocouple

· Matti

It took me a while to find out how to connect MAX6676 to Wemos D1 to read temperature. Many references suggested to use Wemos D1 pins D6-D8 but this fails. Following pinout works:

  • D5 – SCK
  • D6 – CS
  • D7 – S0

Here is the sample code with correct pin definitions:

/*
  Average Thermocouple
  Reads a temperature from a thermocouple based
  on the MAX6675 driver and displays it in the default Serial.
  https://github.com/YuriiSalimov/MAX6675_Thermocouple
  Created by Yurii Salimov, May, 2019.
  Released into the public domain.
*/
#include <MAX6675_Thermocouple.h>
#include <Thermocouple.h>

#define SCK_PIN D5
#define CS_PIN D7
#define SO_PIN D6

Thermocouple* thermocouple;

// the setup function runs once when you press reset or power the board
void setup() {
  Serial.begin(9600);

  thermocouple = new MAX6675_Thermocouple(SCK_PIN, CS_PIN, SO_PIN);
}

// the loop function runs over and over again forever
void loop() {
  // Reads temperature
  const double celsius = thermocouple->readCelsius();
  const double kelvin = thermocouple->readKelvin();
  const double fahrenheit = thermocouple->readFahrenheit();

  // Output of information
  Serial.print("Temperature: ");
  Serial.print(celsius);
  Serial.print(" C, ");
  Serial.print(kelvin);
  Serial.print(" K, ");
  Serial.print(fahrenheit);
  Serial.println(" F");

  delay(500); // optionally, only to delay the output of information in the example.
}

OTA Update for an ESPhome Device

· Matti

After installing the ESPhome firmware to my SH-P01s I quickly noticed I had to update the firmware settings. It turned out that updating the firmware is easy after the device is running on ESPhome.

First I had to find out the IP of the device. I used nmap for this. First I connected my laptop to the same network with the relays and then:

$ nmap -sL 192.168.4.0-255

Starting Nmap 7.60 ( https://nmap.org ) at 2020-02-24 22:11 EET
Nmap scan report for 192.168.4.0
Nmap scan report for 192.168.4.1
….
Nmap scan report for 192.168.4.30
Nmap scan report for relay_1.your.localdomain (192.168.4.31)
Nmap scan report for 192.168.4.32
…
Nmap scan report for 192.168.4.58
Nmap scan report for relay_2.your.localdomain (192.168.4.59)
Nmap scan report for 192.168.4.60
…
Nmap scan report for 192.168.4.255
Nmap done: 256 IP addresses (0 hosts up) scanned in 0.32 seconds

It was easy to find the relay 1 and its IP 192.168.4.31 from the nmap output. Now I could edit the firmware configuration and re-compile the firmware. I used the dockerised version of ESPhome:

docker run --rm -v "${PWD}":/config -it esphome/esphome relay_1.yaml upload --upload-port 192.168.4.31

INFO Reading configuration relay_1.yaml…
INFO Connecting to 192.168.4.31
INFO Uploading relay_1/.pioenvs/relay_1/firmware.bin (428624 bytes)
Uploading: [============================================================] 100% Done…

INFO Waiting for result…
INFO OTA successful
INFO Successfully uploaded program.

Done!

Reflashing Deltaco Smartplug SH-P01 to Work With Home Assistant

· Matti

Note! This document is not valid any more. The SH-P01 devices I purchased on late 2021 were running a patched firmware. Thus, the Tuya-convert is not able to flash the SH-P01s.

This is a documentation of a ongoing work where I’m trying to get Deltaco Smartplug SH-P01 to work with Home Assistant.

Since the Home Assistant does not have a native support for the device I’m planning to:

  • Create an ESPhome firmware with the configuration I found from the Home Assistant discussion board
  • Flash the firmware with tuya-convert
  • Finally, control the device with Home Assistant

Creating ESPhome Firmware

Since I already had Docker installed on my laptop I entered

docker pull esphome/esphome

and got the ESPhome Docker image. After this I created relay_1.yaml with following content:

esphome:
  name: relay_1
  platform: ESP8266
  board: esp01_1m

# Your WiFi SSID and passphrase is defined here
# https://esphome.io/components/wifi.html
wifi:
  ssid: "YOUR_WLAN_SSID"
  password: "YOUR_WLAN_PASSPHRASE"

# Enable fallback hotspot (captive portal)
# In case the device can't connect the host defined above
# it starts to work as an access point with these settings
# https://esphome.io/components/captive_portal.html
  ap:
    ssid: "Smartplug Deltaco 1"
    password: "RANDOM_PASSWORD_NEEDED_IN_CASE_THE_DEVICE_CANT_CONNECT_WLAN"

captive_portal:

# Enable logging
logger:

# Enable Home Assistant API
# https://esphome.io/components/api.html
api:
  password: "RANDOM_PASSWORD_FOR_HOME_ASSISTANT_TO_CONNECT"

# Enable Over The Air update component
# https://esphome.io/components/ota.html
ota:
  password: "RANDOM_PASSWORD_FOR_ESPHOME_OTA_UPDATES"

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO13
      mode: INPUT_PULLUP
      inverted: True
    name: "Deltaco SH-P01 Button"
    on_press:
      - switch.toggle: deltaco_relay_1
  - platform: status
    name: "Deltaco SH-P01 Status"

output:
  - platform: esp8266_pwm
    id: deltaco_smartplug_blue_led
    pin:
      number: GPIO5
      inverted: True
      
switch:
  - platform: gpio
    name: "Deltaco SH-P01 Relay"
    id: deltaco_relay_1
    pin: GPIO12

light:
  - platform: monochromatic
    name: "Deltaco SH-P01 blue LED"
    output: deltaco_smartplug_blue_led

This was taken from the discussion board referred above. All I needed to do was to edit the SSIDs and passwords and compile a new firmware:

docker run --rm -v "${PWD}":/config -it esphome/esphome relay_1.yaml compile

This printed a lot of debug information and ended with following lines:

Building .pioenvs/relay_1/firmware.bin
 Retrieving maximum program size .pioenvs/relay_1/firmware.elf
 Checking size .pioenvs/relay_1/firmware.elf
 DATA:    [=====     ]  45.3% (used 37148 bytes from 81920 bytes)
 PROGRAM: [====      ]  41.5% (used 425048 bytes from 1023984 bytes)
 Creating BIN file ".pioenvs/relay_1/firmware.bin" using ".pioenvs/relay_1/firmware.elf"
 ========================= [SUCCESS] Took 29.83 seconds =========================
 INFO Successfully compiled program.

And yes, the compiled firmware was at .esphome/build/relay_1/.pioenvs/relay_1/firmware.bin! All I needed to do is to get this firmware to the device.

Flashing the Firmware with tuya-convert

The tuya-convert offers a docker image to make the flashing, but I could not make it work. Therefore I installed the script and required packages directly to my Ubuntu as instructed in the README:

git clone https://github.com/ct-Open-Source/tuya-convert
cd tuya-convert
nano -w config.txt
# Changed value WLAN to my device name, wlx7cdd901255af
# Enumerate your device names with "ifconfig"
sudo ./install_prereq.sh

At this point I disconnected by WLAN from the local access point so it was available for tuya-convert. I also copied the ESPhome firmware.bin to tuya-convert/files/relay_1.bin. Then it was time to start the flashing script:

sudo ./start_flash.txt

When instructed I joined my phone to new WLAN access point “vtrust-flash” and after this was done I reset my SH-P01. It took me a while to understand why it did not respond when I pressed the button for the magic 6 seconds. Finally it turned out that the device already had connected to the AP.

The script did the trick. First it downloaded the firmware backup and after that it asked which binary file I wanted to install. After a while the relay_1.bin was uploaded.

Installing Home Assistant

Since I have a Xen server I installed the Home Assistant to a fresh VM running Debian Stretch. The Home Assistant configuration was quite uneventful as everything went as planned.

Since the server and the SH-P01 were on the same network segment (same LAN) the switch was discovered right away. All I needed to do was to give the API password I defined in relay_1.yaml and wrote to the ESPhome firmware.

Everything worked: the relay, the light and the button.

Flashing More Devices

To add more devices just make a copy of the ESPhome configuration (relay_1.yaml) and edit the esphome:name -setting:

esphome:
  name: relay_2
  platform: ESP8266
  board: esp01_1m

The device names must be unique and they’re used for example as a hostname of the device. Home Assistant creates its ID:s from device names (e.g. “Deltaco SH-P01 Relay” becomes “switch.deltaco_sh_p01_relay”). In case you’re worried about these entity ID:s you might want to give distinctive names to avoid overloading (“switch.deltaco_sh_p01_relay_2”, “switch.deltaco_sh_p01_relay_3” etc.). You might also want to change some of the passwords. After this just repeat the steps with the new firmware.

The sample configuration toggles the relay when the button is pressed. If you don’t want this behaviour but want to control the relays by the Home Assistant automations just remove this codeblock:

on_press:
  - switch.toggle: deltaco_relay_1

Starting MariaDB on Ubuntu Bionic causes timeout

· Matti

Starting MariaDB 10.1 causes timeout. Syslog says:

Apr  5 20:55:53 megis systemd[1]: Starting MariaDB 10.1.38 database server...
Apr  5 20:55:53 megis mysqld[6892]: 2019-04-05 20:55:53 140074458918016 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) starting as process 6892 ...
Apr  5 20:55:54 megis kernel: [  980.249533] kauditd_printk_skb: 2 callbacks suppressed
Apr  5 20:55:54 megis kernel: [  980.249535] audit: type=1400 audit(1554486954.089:38): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=6892 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=121 ouid=0
Apr  5 20:57:23 megis systemd[1]: mariadb.service: Start operation timed out. Terminating.

There can be more lines where AppArmor says

audit(1554487043.725:40): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld"

Some notes in the net suggest that you should set the TimeoutSec=infinity in the systemd configuration, but this did not help in my case. I had to disable the AppArmor for /usr/sbin/mysqld. It wasn’t enough to put it to complain mode. Here are the instructions:

Get apparmor-utils

If you can’t execute “sudo aa-disable” you have to install the AppArmor utilities:

sudo apt install apparmor-utils

Create AppArmor profile for MariaDB

By default the AppArmor profile for /usr/sbin/mysqld is empty which causes “aa-disable” to fail. Add following lines to “/etc/apparmor.d/usr.sbin/mysqld”:

/usr/sbin/mysqld {
}

Disable AppArmor for /usr/sbin/mysqld

After this say:

sudo aa-disable /usr/sbin/mysqld

Writing keyboard layout for xorg – Notes

· Matti

I had a privilege to be the first person in universe to implement Skolt Sami keyboard for Linux. I had two references:

The job was pretty straight-forward: change the keyboard combinations in the file.

Keycodes

The idea of xkb file is to translate keycodes to characters. The first step is to identify the keys in the keyboard. Each key produces its own keycode. For a start I used Figure 2 in this documentation to get an idea ofo the keycodes. However, my keyboard layout (and the documented Sami layout) differed from fig 2.

To get a keycode for a key:

$ xev -event keyboard

KeyRelease event, serial 28, synthetic NO, window 0x2200001,
 root 0x12d, subw 0x0, time 85909731, (-350,-52), root:(244,267),
 state 0x0, keycode 38 (keysym 0x61, a), same_screen YES,
 XLookupString gives 1 bytes: (61) "a"
 XFilterEvent returns: False

The xev utility gives you the keycodes but this code cannot be used as such in your xkb definitions where the keys are identified with a hex tuplet (e.g. AC01) or a tag (e.g. LSGT). To get your xkb-compliant key identifier:

$ xkbcomp :0 keyboad-mapping.txt
$ grep 38 keyboard-mapping.txt
 <AC01> = 38;
 <PROP> = 138;
 <I238> = 238;
 key <I238> { [ XF86KbdBrightnessUp ] };
 { [ 38, 18 ] },
 left= 382;

Wow! Now I know that A in my keyboard -> keycode 38 -> AC01 in xkb definition.

Characters not defined in keysymdef.h

Most of the xkb howtos I browsed explained that all the symbols are defined in the keysymdef.h. Well, that wasn’t the case with Sami characters. Instead of using the defined keyboard symbols I had to use Unicodes. Say we want to implement Đ character.

  1. If possible, try to get the character to a clipboard. I ended up using Windows implementation to get all three different hyphens right.
  2. Go to http://www.utf8-chartable.de/unicode-utf8-table.pl and locate the character there. This evil beast was hiding in the page “Latin Extended-A”.
  3. Get the Unicode code point (for Đ it is U+0110) which can be used in the xkb definition (“U0110”).

When implementing my second keyboard layout I found a search tool which helped me a lot in finding exotic characters. Workflow:

  1. Start a Windows with selected keyboard layout in a virtual machine. Start a Notepad.
  2. Push a key to see what character appears to the notepad. Copy and paste it to search tool.
  3. The Unicode number can be entered directly to the keyboard definition but I tried to find the symbol from the keysymdef.h whenever I could. It makes keyboard definition file more readable.
  4. Repeat with Shift+Key, AltGt+Key and Shift+AltGr+Key.

VoidSymbol

To avoid strange characters appear when you type the undefined combinations use “VoidSymbol”. So if you define keys like this:

 key <AC01> { [ a, A ] };
 key <AC02> { [ s, S ] };

and press AltGr+A or AltGt+Shift+S you might get some random characters To avoid these chars use VoidSymbol to make sure that no definions are inherited (or whatever neat feature causes this):

 key <AC01> { [ a, A, VoidSymbol, VoidSymbol ] };
 key <AC02> { [ s, S, VoidSymbol, VoidSymbol ] };

Using xrdp Desktop for Everyday Life (Ubuntu Xenial)

· Matti

Preface: I Need the Latest xrdp

During the holidays the keyboard of my laptop started to indicate that there is no eternity in the world. And especially not among the cheapo laptops. Since a new laptop with a decent screen and processor would const a fortune, I decided to try another approach. Why don’t I install a desktop Linux to my Xen server and use it for daily tasks? In this setup the laptop wouldn’t need to have a capable processor.

I wanted a setup where the connection would be direct without any servers between. I don’t want anyone to follow my mediocre node.js stumbles or follow my bank sessions. TeamViewer and similar helpdesk tools are no-go.

I did some testing with NoMachine NX, but it felt a bit clumsy although the setup was painless. Using X was not sufficient option since you want to hear the sounds when watching YLE Areena, right?

Finally I ended up working with xrdp, a Linux realisation of Microsoft’s Remote Desktop Protocol. The protocol supports sound and its performance should be enough for multimedia. Well, it quickly turned out that the pre-packaged version of xrdp (the server) did not have support for sound. It didn’t support mouse wheel either which is another “must”. Thus I needed to compile the latest version myself.

The Beef: Compiling xrdp With Sound

Compiling xrdp

Get the source from xrdp site. My version was 0.9.4. First compile xrdp. Make sure you have build-essential installed and give it a try:

cd xorg/
./configure
make
sudo make install

Compiling X11 with RDP support

It turns out that the pre-packaged Xorg does not have RDP support and we need to compile X11 with RDP support as well:

apt-get install xsltproc flex bison libxml2-dev python-libxml2
cd xorg/X11R7.6/
./buildx default

Install (copy) the binary to /usr/local/bin as root:

cp rdp/X11rdp /usr/local/bin

Now start (again, as root):

xrdp-sesman
xrdp

By default xrdp server supports multiple session types. Since I want to provide the RDP protocol only I have to edit /etc/xrdp/xrdp.ini. The session types are in the bottom (after [Channels] section).

Leave only X11rdp connection type:

[X11rdp]
name=X11rdp
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=-1
xserverbpp=24
code=10

Getting sound to work

Building Pulseaudio sound modules requires a bit extra work.

First, get the version of your Pulseaudio installation:

$ pulseaudio --version
pulseaudio 8.0

Get the pulseaudio source and unpack it somewhere. I used ~/src/pulseaudio and now I have ~/src/pulseaudio/pulseaudio-8.0. Now go to forementioned directory and compile your Pulseaudio:

apt-get install intltool libjson-c-dev libsndfile1-dev
./configure --without-caps
make

Now back to xrdp:

cd xrdp/sesman/chansrv/pulse

Edit PULSE_DIR (in xrdp/sesman/chansrv/pulse/Makefile) to point to the path you have your Pulseaudio source. In my example:

PULSE_DIR = /home/matti/src/pulseaudio/pulseaudio-8.0

Then simply:

make

Now you have module-xrdp-sink.so and module-xrdp-source.so. Copy them to Pulseaudio libs (as root)

cp *.so /usr/lib/pulse-8.0/modules/

Add following lines to your /etc/pulse/default.pa:

load-module module-xrdp-sink
load-module module-xrdp-source
set-default-sink xrdp-sink
set-default-source xrdp-source

Note! In my case pavucontrol crashed the RDP connection for whatever reason. If you encounter problems you can list sinks and sources with these commands:

$ pacmd list-sinks | grep -e 'name:' -e 'index'
 * index: 1
 name: <xrdp-sink>
$ pacmd list-sources | grep -e 'name:' -e 'index'
 index: 1
 name: <xrdp-sink.monitor>
 * index: 2
 name: <xrdp-source>

You can restart the xrdp server by restarting xrdp and xrdp-sesman.

Linux Client

At this point your Linux desktop can be connected using Windows Remote Desktop Client. Since none of the pre-packaged rdp clients did not work, had lag or did not support sound i decided to build latest FreeRDP. I personally decided latest nightly build offered from their CI at https://ci.freerdp.com/job/freerdp-nightly-binaries/.

To open a connection I enter:

/opt/freerdp-nightly/bin/xfreerdp /kbd:0x0000041D /sound /v:[MY SERVER NAME]

The /kbd value uses Swedish keyboard layout. You can get the possible values by entering:

/opt/freerdp-nightly/bin/xfreerdp /kbd-list

You don’t need to edit the keyboard settings in the target machine. Actually editing them made FreeRDP to segfault.

Epilogue: Was it Worth it?

The original goal was to use the remote desktop as a workstation for daily use: browsing the Net, reading emails and doing some garage programming. Was this setup good enough for this?

No.

There is slight lag with the display: when you close a window it can take 1-2 seconds before the screen updates. Changing the virtual desktop is sluggish. The mouse roller is inaccurate. I used this for one or two days and then destroyed the whole thing.

Acknowledgements

Reading PDF fields with Python/pdfminer

· Matti

pdfminer is a PDF data extraction class written completely in Python. You can use it to extract data from PDF fields as well. However, doing so can be a headache since the form entries may have child objects which you should search as well. Most of the sample codes I found from the net did not do this properly or there were problems with the encoding of the strings.

Here is the sample code I wrote to demonstrate getting the data:

from argparse import ArgumentParser

from pdfminer.pdfparser import PDFParser 
from pdfminer.pdfdocument import PDFDocument

from pdfminer.pdftypes import resolve1, PDFObjRef

def load_form(filename):

    """Load pdf form contents into a nested list of name/value tuples"""

    with open(filename, 'rb') as file:
        parser = PDFParser(file)
        doc = PDFDocument(parser)
        return [load_fields(resolve1(f)) for f in
                   resolve1(doc.catalog['AcroForm'])['Fields']]
                   

def load_fields(field, parent_var=None):

    """Recursively load form fields"""

    def escape_utf16(param_str):
        if type(param_str).__name__ == "PSLiteral":
            param_str = str(param_str)
        if isinstance(param_str, basestring) and param_str[:2] == "\xfe\xff":
            # If we have string with UTF-16 BOM remove BOM and null characters
            param_str = param_str[2:].translate(None, "\x00")
        if isinstance(param_str, basestring):
            # Encode all strings to UTF-8 (PDF uses ISO-8859-15)
            return param_str.decode("iso-8859-15").encode("utf-8") 
        return param_str
        
    form = field.get('Kids', None)

    if form:
        # This is a child form, recurse into
        new_parent = field.get('T')
        if parent_var:
            new_parent = parent_var+"."+new_parent
        return [load_fields(resolve1(f), new_parent) for f in form]
    else:
        # Some field types, like signatures, need extra resolving
        if (parent_var):
             return (parent_var+"."+field.get('T'), escape_utf16(resolve1(field.get('V'))))
        else:
             return (field.get('T'), escape_utf16(resolve1(field.get('V'))))


def flatten_form (deep_form):
    """ Flatten given form (from load_form()) to a dictionary """

    dict_form = {}
    
    for this_item in deep_form:
        if isinstance(this_item, list):
            this_flat_item = flatten_form(this_item)
            for this_key in this_flat_item.keys():
               dict_form[this_key] = this_flat_item[this_key]
        else:
            dict_form[this_item[0]] = this_item[1]

    return dict_form
    
def parse_cli():
    """Load command line arguments"""

    parser = ArgumentParser(description='Dump the form contents of a PDF.')

    parser.add_argument('file', metavar='pdf_form',
                    help='PDF Form to dump the contents of')

    return parser.parse_args()



def main():
    args = parse_cli()

    # Read form
    form = load_form(args.file)
    # Make a "flat" dictionary from the form data given by load_form()
    form_flat = flatten_form(form)
    
    # Print form data
    form_keys = form_flat.keys()
    form_keys.sort()
    for this_key in form_keys:
        if isinstance(form_flat[this_key], basestring):
            print this_key+": "+form_flat[this_key]
        elif form_flat[this_key] == None:
            print this_key+": None"
        else:
            print this_key+": Unprintable"
    
if __name__ == '__main__':
    main()