Fender Standard P-Bass – Pickup Swap

As I mentioned in my previous posts about my 2012 Mexican Fender Precision I had recently bought and set up, a pickup swap was always on the cards.

What I wanted was something that sounded like a “real” USA P-Bass from the ’70s – the sound of Heavy Rock and Punk / New-Wave when I was growing up1. This shouldn’t be difficult in principle as the ’70s was before all the aftermarket pickup options anyway, so the sound in my head was almost certain to be coming out of a normal mass-produced stock pickup made by Fender.

I theoretically wasn’t in a hurry, but I thought I’d do some research and see what people replace the pickup in a MIM precision with and… oh dear. This search leads down a twisty tunnel of infinite choice where there are hundreds of options and no-one agrees on anything2. Ok, this was not going to be easy 😉

Fortunately a bit lot more internet searching reveals that the choices divide into 3 groups:

  • Sounds exactly like an “authentic” vintage pickup from ’58, ’62, ’63 (or whatever).
  • Sounds like an “authentic” vintage pickup from ’58, ’62, ’63 (or whatever), but better!
  • Sounds like a huge rock juggernaut that will crush everything in its path, destroy worlds, as hot as the centre of the sun etc. etc. etc.

This made things easier as better = more than I wanted to spend and huge = not what I was after, so I could eliminate those. The exactly category was a broader church that included a few cheaper pickups that people recommended and I wasn’t looking for a perfect facsimile anyway, so no point paying a fortune3.

I eventually came up with three candidates: Seymour Duncan SPB-1, Fender “Original Vintage” and EMG GZR. These were all in the £65 – £90 price range and there were good-sounding demos on YouTube of all three. None of them stood out though so it wasn’t an obvious sonic choice. In the end I chose the Fender, mostly because if anyone can make a Fender-sounding pickup it should be Fender themselves, but also because it came with a period-correct brass shielding and mounting plate and the spacer foam, so should be an easy replacement.

On to the bench and off came the scratchplate.

The cavity was shielded with a thin layer of conductive paint and the controls are all full-size with a decent jack socket.

All the pot fixing nuts were a bit loose and there was remnants of the plastic film that the pick-guard was covered with under them, so I removed that and tightened everything up properly.

The 50nF capacitor was a tiny ceramic disc, these aren’t great components4 and I had some proper 47nF film capacitors5 so I immediately replaced that. This will not make a significant difference to the sound, but it’s a small upgrade.

The next step was to snip the wiring, remove the old pickup then fit the new one. That is simple, although I had to loosen the strings off a bit to get the new pickup in as the brass grounding plate meant I couldn’t wiggle the two halves of the pickup around to slide under the strings.

I test assembled the new pickup and got the scratch-plate in position to check the fit and position of the mounting screws. Everything lined up fine after a bit of jiggling about so I temporarily put it together minus most of the pick-guard screws and gave it a test play to check the wiring was OK and I could adjust the heights properly.

Hmmm… it worked, but there were two small problems. The springy mounting foam wasn’t thick enough to get the pickups high enough and the magnet pole pieces stuck out proud of the covers. The sticking-out magnets are bad news as I am quite heavy-handed with my picking hand and it is possible for the strings to strike the magnets. This makes a horrible clack! sound, so I had to fix that sharpish.

Fortunately there is an easy fix to both problems. A couple of small rectangles of 4mm plywood made packers underneath the brass baseplate to raise the pickup enough to give the proper range of adjustment6.

Then taking a couple of sections cut from a cable tie and fixing with double-sided tape inside the pickup cover raises it just enough to drop the pole pieces  below the level of the cover (the picture shows that on the fitted cover).

Perfect! I can adjust the height to where it needs to be and there is no danger of the strings striking the magnets.

Re-fit the scratchplate, tweak the pickup height and off we go…

Now, this is the point where most people get over-excited and sing the praises of the new pickup, how it has transformed their life and saved their first-born etc. I’m not that daft, and I also took the precaution of making direct to PC recordings of the old pickup so I could compare it with recordings of the new one and not just rely on an imperfect memory.

So, how does it compare?

It’s pretty similar, the old stock ceramic pickup wasn’t bad at all. There are two main improvements I can hear though; the new pickup is clearer and slightly more “detailed” sounding7 and the balance between the strings is better with a more forward and present high end8. I’m not sure this would all be very audible with a loud band behind you, but it does sound a bit better in a close listening situation or on a recording. I’d say it’s enough of an improvement to be worth the £79 delivered that the pickup cost me. I’d strongly suspect that spending much more on a high-end boutique pickup wouldn’t yield any further improvement though; certainly not worth £40 – £100 more.

The Mexican P-Bass Ceramic Pickup

Interestingly, I checked the resistance of the old ceramic pickup in circuit (4,700 ohms) and the resistance of the new alnico5 pickup (10,900 ohms) so I could compare them. The old ceramic pickup has slightly higher output, so higher resistance does not necessarily equal higher output – it’s more complicated than that. Wire gauge, bobbin design, magnet type and magnet strength all play a part too in deciding the characteristics of a pickup, so beware of simplistic comparisons!

Altogether a successful exercise in upgrading and it has made my P-Bass a little more authentic 🙂

 

 

 

 

 

Fender Standard P-Bass Setup – Part 2

The Bench & a pile of tools…

Real life got in the way for a few weeks and I didn’t get the uninterrupted time I needed to do the fretwork until a bit later. I knew this would take at least 3 hours and I didn’t want to get into it and have to stop… but at last I had time.

So, what needs doing? – three problems:

1) Rough fret ends.

2) A couple of high frets.

3) Top two nut slots a touch high.

Only the first thing badly needed doing, the others could have been left if I was short of money and needed to pay someone else to do the work. That is why it is nice to have the skills and tools to do all the work yourself, it’s just having the time needed to do the job as the materials cost is tiny.

 

I like to do a “map” of the fretboard high/low spots first just so I see what I am looking at and get an idea of the amount of work. Here it is:

Fret 19 was the worst and fret 18 was also high so it would be a bit chancy just shaving the individual frets down, it is easy to end up playing whack-a-mole chasing the high spot between frets. It’s more efficient and safer to do the whole lot with a levelling beam and get everything dead level and consistent. I was going to have to polish the frets anyway as fixing the sharp ends and doing a slight fretboard edge roundover would involve sanding and polishing the end part of the frets, may as well run the beam over the lot and do a complete job.

First task was to sort out the rough fret ends by filing them smooth. The little Stewart McDonald fret end dressing file is the best one I have found as it is small, precise and a fine cut. There is one safe side that is rounded with no teeth and this rides against the fretboard to avoid digging in and damaging it.

Unfortunately the numpties at Fender had decided to part-lacquer some of the rosewood edge of the fingerboard (and not do a very good job) so my little file safe edge started to flake this lacquer off, but only at the headstock end where it had ended up much too thick. This looked terrible!9

The “After” picture… they’ll never know 🙂

After a bit of experimenting with using a single-sided razor blade as a scraper I found that this excess lacquer came off quite easily leaving a nice smooth rosewood finish underneath and I only had to part-scrape the first 5 frets on either side anyway. I had placed masking tape along the line between maple and rosewood to make sure that I didn’t touch the neck, so this made a nice line to scrape up to10.

Having solved the lacquer drama I carried on with my little file taking all the rough fret ends off. This is pretty quick and you can immediately feel the massive difference it makes. The ends had already been part-filed at the factory when doing the bevel, so it didn’t need a lot of work, just the sharp edges softened.

Next step was to smooth the filed ends with various grits of sandpaper to make them really nice. I also wanted to do a slight round-over of the fretboard edge as it was really a dead right-angle, so felt a bit uncomfortable under the fingers. There is not much information on-line about ways to do this with an already fretted neck and I wasn’t that happy with my previous test attempts on various scrap necks11.

Fortunately, I had recently seen a Crimson Guitars video on YouTube which demonstrated a technique that I had never seen before to do a combined fret-end polish and slight fretboard edge bevel. I wanted to keep the round-over mild so I skipped the levelling beam step (and the filing part that I’d already done) and went straight to the 240 then 320 grit tightly-rolled sandpaper cylinders (3:45 onward in the video) and then the higher grit papers and fret erasers to do the polish. This worked great and gave a really nice result with a lot less chance of going wrong than the other methods I had looked at and tried. Tick V.G. to Crimson.

Next up, the fret level. As I said earlier, I could have got away without this as it wasn’t buzzing, but the fret end finishing had left the ends of each fret really shiny and it looked odd with the rest of the fret quite dull. I’d have wanted to do a full fret polish anyway to make them all look nice, so may as well level first.

First step is to get the neck dead straight and flat by adjusting the truss rod with a notched straight-edge on the fretboard. This was really easy on this neck, it went flat as soon as the truss rod was backed off – always a good start. I re-checked with the fret rocker with the neck flat to see if any of the high frets had changed in any way; nope, exactly the same.

400mm Levelling Beam

Next the tops of each fret were marked with a black sharpy so the effect of the beam could be seen. I usually use 320 grit paper on the beam as it cuts quite slowly so is quite forgiving. This levelling went pretty quickly – most of the frets were level anyway so the beam just kissed the tops. I had to do a bit more work on the two high frets to get them level, but still not too bad12.

Then I re-marked the fret tops with a sharpy and crowned with a 300 grit diamond crowning file – also very quick as most frets hadn’t got a flat top so only needed a few strokes of the file.

The next bit is the very tedious process of polishing each fret by working up through the grits of sandpaper from 400 to 250013. I tried a new method and started out doing each fret individually using a metal fret protector as I have seen some people do, but this was so fiddly and annoying that I quickly gave up and went back to my normal method of taping up the whole fretboard with masking tape so I could use a foam pad with the sandpaper round it and run it along the whole fretboard in one go. Much quicker!

Looking nice now!

Once I had finished the polish I took off the tape and cleaned the fretboard with naptha14 to get any dirt or sandpaper dust off and then finally applied a conditioner to the rosewood. I have started using Howard “Feed-n-Wax”, an American product that I saw recommended by one of the Luthiers on YouTube.

I really like it as it very slightly darkens the wood and leaves a nice smooth slick finish15.

This fretboard was quite light in appearance before I conditioned it, it actually looked a lot like the new Pau Ferro fretboards that Standard P-Basses are made with post the CITES regulations. I think it looks nicer slightly darker, so I was pleased with the effect.

 

 

Nut files – 48 / 65 / 85 / 105

The final step was to sort out the high nut slots on the D and G string.

Now; I always find it quite nerve-wracking cutting the nut slots. It is very easy to over-cut the depth and write the nut off 16, so I am always very careful. I used my 65thou and 48thou nut files to deepen the slots slightly and I stopped as soon as a 22thou feeler gauge was a sliding fit.

That was enough to play comfortably, but still a conservative clearance, so no danger of the dreaded 1st fret buzz17.

 

And that was it – job done. Put the bass back together, set it up again and it’s ready to go. I dropped the action down to 5/6418 (E) to 4/6419 (G) while I was at it and the fretwork handled that no problem20.

I’ve got some new strings ready to go on when I get round to it and I’ll tweak the intonation then and go back over the setup to get it just right.

Splendid!

Fender Standard P-Bass Setup – Part 1

2012 MIM P-BassAfter playing bass for years, but never owning an actual Fender P-Bass1 I recently got this nice clean 2012 Candy Apple red Mexican Standard P-bass second-hand for very sensible money.

I wanted a MIM P-Bass as it is much cheaper than the US models and also easy to modify. It doesn’t deviate much from the “standard” 1960s design, so almost all the aftermarket parts just fit. Post-2008 is also a good period for Mexican Fenders as the designs got pretty close to the US instruments and the quality improved too.

It was owned by someone that had hardly played it or messed about with it, so it was pretty much in mint condition and completely stock specification.

Over the years, due to lack of adjustment, the neck had got rather bowed with a massive relief (over 40thou) and the overall action was very high, over 9/64in at the 17th fret. So I had to fix that so it would at least play before doing anything else.

After about 2/3 of a turn of the truss rod and a bridge saddle adjustment I got it to 12thou relief and 6/64in (E) – 5/64in (G) action (Fender stock suggestion). The intonation was very close already so I didn’t touch that. The first fret action was a bit high on the highest 2 strings – not painful, but 30thou-ish instead of 20-22thou so a tiny bit stiff. I went with it for now.

Result – it plays. The sound is pretty respectable on just the stock pickup, so my plan of doing an immediate pickup upgrade is not the priority I thought it would be2.

The only serious snag as far as I am concerned is that the fret ends are a bit too rough to be comfortable when sliding my hand up and down the neck. It’s the G-side where it shows up mostly, the frets don’t stick out proud of the fretboard edge, but the bevel isn’t smooth at all. I also quickly went over the frets with a rocker and there were 2 up the dusty end that were a touch high, not high enough to buzz at the action I had set, but they would be a problem if I tried to go much lower than my usual action for any reason.

Ironically, it’s the fact that this bass is almost unplayed that is the problem, normally a 5-year old bass would have seen a guitar tech at some time in it’s life and the neck issues would have been sorted out with a decent fret dress and thorough setup (£95-ish maybe outside of London).

Now, at this point on the discussion forums everyone piles in with “Mexican Fenders are rubbish, should have bought a US Fender…” etc. This kind of misses the point that I would have had to pay at least 2-3x as much for a s/h USA P-bass and it is still not guaranteed to be perfect3. The cost of a MIM P-Bass and a setup is still far cheaper than a USA P-Bass and the MIM instrument is a great modding platform. Yes, the US P-Bass has slightly better components4, a better finish and so on, but the MIM bass can play just as well as the US bass when the setup work is done.

Fortunately, I do all my own instrument teching and I have all the tools to do fretwork. There is absolutely nothing wrong with this bass that I can’t fix.

To the workbench!

 

 

 

 

 

 

 

TO BE CONTINUED…

Fixing a Watchman Plus Oil Level Monitor

After the recent round of enjoyable bass guitar fettling a much more mundane thing needed fixing.

The little gadget in the pic is the monitor unit for a Watchman Plus domestic heating oil tank level monitor. This is a little module that looks something like a wall-wart mains power supply. On the top is an antenna that communicates to a sensor unit that is buried inside a domestic heating oil tank reporting back the oil level.

This is quite an old and obsolete model now, but it has worked perfectly for years until this summer when it started periodically losing the tank signal (the little red led at the bottom right flashes and the display indicates “r”).

Step one was replacing the battery pack in the tank transmitter as it was at least 5 years old and that is around the usual battery life. This was quite fiddly and awkward to do as the pack was buried in between the inner and outer shells of the tank and could only be reached blind at arm’s length. The results were promising at first, but after a few hours the signal was being lost again. Very annoying!

Especially annoying since the transmitter battery pack was about £331 and I really didn’t want to fail to fix it having spent that much.

Time to get dismantling! The pic on the left is the circuit board after I had extracted it from the plastic case by levering the case open2.

Now what?

Fortunately, I have a superpower3. My first career was in Electronics design and I specialised in embedded systems. What that means is that I can look at a gadget like this and work out enough about it to deduce how it works and stand a good chance of fixing it as long as it isn’t completely toasted.

The first thing that struck me was that there was no mains transformer or evidence of a switch-mode power supply. Oh, oh – that means a capacitive reactance voltage dropper … which also means the circuit is live to mains with no isolation. That makes the board pretty dangerous to work on and hard to test as there is no simple way of getting a ground reference to measure things4. My thought was to try and trace the power supply part of the circuit and see if I could reason about the fault based on the failure modes I was seeing5.

This is the power supply circuit:

Because the circuit is mostly surface-mount it is pretty hard to work out the values of some of the components, or even what kind of components they are. That is where the professional experience comes in – I know what type most surface-mount components are just by the look of them and I can reason out what the mystery ones are based on how the circuit must work6. There are various capacitors that have mystery values, but they are not important in determining how the circuit works.

The key component in this circuit is C1, a 330nF mains rated (X2) capacitor. In normal operation this presents a capacitive reactance of around 10k7 at 50Hz meaning that very roughly 24mA flows through it and is split between the zener diodes and the current drawn by the circuit through the 5V voltage regulator. That is pretty consistent with the PIC microcontroller and a 433MHz receiver module – I’d bet they take 10mA-ish @ 5V.

So far, so good. I’d already had a bit of a google for failures of these oil monitor units and I found a link8 suggesting that the mains rated capacitor can fail. This was enough of a clue that I went looking for failure modes of mains rated metal film capacitors. They couldn’t be failing open-circuit as the unit still worked intermittently, nor could they be failing short-circuit as this would cause a large over-current and blow the unit to bits.

It turns out that the clue is in the “X2” rating which means that the capacitor can survive over-voltages and is safe to connect directly across the mains between Live and Neutral. The way it survives these over-voltages is by self-healing, which crudely means that if the insulation between the capacitive plates fails, the current that flows vaporises the metal film immediately around the failure. This cunningly clears the short-circuit and the capacitor continues operating. Sneaky.

However, there is a snag. If this happens often enough, quite a lot of the metal plate material can be lost resulting in the capacitance value getting smaller as time goes on. If this happened the capacitive reactance would rise, and if the capacitance got too low there would not be enough current flowing for the voltage regulator to work properly if the mains voltage dropped slightly – the 5V would dip9 and cause the microcontroller to reset. This was exactly what I was seeing happen.

So, enough theory – let’s remove the 330nF capacitor and see what it measures…

…and there we have it, 0.177uF10 – just over half the proper value. That capacitor is stuffed!

The rest is routine – buy and solder in a new 330nF X2-rated capacitor (22.5mm radial pitch, costs 29p from RS components), stick the case back together and the monitor is repaired.

I tested it in situ and it ran happily indicating the tank level for a weekend without resetting or doing weird things.

Fixed!

 

 

(See here for someone else who found the same problem in a little mains timeclock and made an interesting video about it.)

Fixing my Yamaha BB1100s Bass (Part 2)

This is the part 2 of resuscitating my old Yam bass that was in a rather filthy and corroded state.

So, because I’m not quite as exacting with my own guitars as I would be with someone else’s and only had a few hours to get this thing playable again, I took a shortcut in cleaning the frets. If I was being thorough I would have taken the neck off, adjusted the truss rod until the neck was dead straight, checked all the frets for levelness1. Then (probably) skimmed them with a levelling beam, re-crowned all the frets, masked the fingerboard with masking tape and polished the frets using wet&dry sandpaper in progressive grits from 400 to 2500, then micromesh and metal polish until I could see my face in the frets. Then cleaned and oiled the fingerboard.

It would have been lovely… but this takes about 3 hours to do properly and I didn’t have time. Pragmatism rules – so off came the strings and I used a metal fingerboard protector and polished a fret at a time with 1200 grit wet&dry. Then cleaned up the fingerboard with naptha2 to get the worst of the gunk off – and voila, the picture shows the result. It’s not perfection, but it’s easily good enough to play again3.

 

Next step is to restring it and get the neck relief sorted out.

The pic shows the truss rod adjustment with the neck screws slackened and the neck tilted up. It’s a M6 threaded truss rod with a buried nut in the neck heel that takes an 8mm socket. Theoretically this should be easy to adjust given the right tool – practically, it is a bitch because the rod sticks out of the nut a long way so it needs a (slightly) special tool that comes with the bass …but I never had this tool.

I had bought an 8mm truss rod socket tool to do the job, but it wasn’t deep enough until I drilled it out on a bench drill. That fitted, but only when the neck was tilted out of the pocket – hence the picture. That sucker took some cranking to flatten the neck, probably a turn or so by the time I had finished getting the relief down to 13thou4.

The rest of the action setup was pretty straightforward, I went with the Fender P/J-Bass “standard” numbers of 5/64in5 on all strings at the 17th fret. The first fret action was 20 – 22thou6 which is pretty much perfect, so the nut didn’t need touching.

So that was the basic playability sorted, light-ish, but nothing buzzes at my level of fumbling7. Result.

Next step – plug it in and make a noise … ouch! Not so good.

 

Snap, crackle and pop! The controls were very noisy so I had the back control plate off and squirted contact cleaner into everything and twisted the pots back and forth to clean all the dust and gunge out. That fixed it.

This is actually a switchable active / passive bass, but I prefer the passive sound so I didn’t bother putting a battery in and testing the active circuit. It worked last time I used it.

I’m not completely convinced the jack is making the best contact possible, so that may need to be replaced at some point. It’s OK for now though. Switchcraft replacements are about a tenner via Allparts.

 

The last problem was the tuning machine heads were really loose and prone to slip. These are special Yam tuners with friction tension collars that can be adjusted to set the “stiffness” of the tuners. As usual, these need a special tool to adjust …which I never had.

This tool is like gold-dust, people are always asking where to get them from on the vintage Yamaha discussion groups, but they can’t be bought so the only solution for me is to bodge it8. Molegrips with a bit of thin rubber on the jaws to avoid chewing the collar up was the dirty solution. Job done.

 

…and she plays a treat! Bassy goodness all round.

I’ve got a new set of strings to put on, the 45-105 round-wounds on there are very dead and a bit manky. I might also polish the frets a bit better and oil the fingerboard when I change them, but there’s no hurry now it plays properly again.

 

Fixing my Yamaha BB1100s Bass (Part 1)

After a bit of a work-imposed gap in hobby activities, one of my oldest pastimes has reared it’s ugly head again – music. I have been playing guitar and bass guitar for years, never getting very good, but enjoying myself and even playing in the odd band or two over the years. (on Bass)

My one bass, a late 80s Yamaha BB1100s, has been sitting in an unheated warehouse at work for almost 10 years. The poor old thing has been very unloved, which is a shame as it is a really nice instrument. I bought it s/h in preference to a Fender USA Jazz bass or P-Bass as it simply played and sounded better to me. Of course, a Fender of that era would have appreciated in value where my old Yam is more or less worthless9. Bah! I like it anyway.

So, what is wrong with it? Er … look at those frets:

They appear to have gone rather green and the fingerboard is filthy. In fact the whole thing is pretty grubby and rather …sticky. (where did I last play this???)

The neck is also bent like a banana. What should be a small amount of neck relief at maybe 12-14thou10 is more like 40thou11 at the 7th fret so it is clear the truss rod is waaaay out of adjustment. In fact, the playing action is massively high. Did I ever set this thing up? Not sure I even knew how to do that the last time I was playing this seriously. Or did it have uneven frets and buzzed if the action was set to a normal height??? Can’t remember.

So, it’s going to need some serious work to get this old thing playing again. Fortunately, I’ve learned a thing or two about guitar setups since then so I’m not daunted!

To be continued…

Message Notifications using PHP and Pushbullet

I’ve been amusing myself following through an excellent series on self-hosting WordPress by Ashley Rich. Mostly so I can experiment with Nginx, pinch the best ideas and use them myself 🙂

Everything had gone swimmingly on a test VPS running the latest LTS version of Ubuntu, but when I tried to implement some of the ideas on the shared hosting1 where this blog runs I ran into a bit of trouble. I was trying to get WordPress installation checksum verification2 and push messaging via Pushbullet working, but the https connect to the Pushbullet API would always fail. After some experimentation it turned out that the command line version of Curl was ancient3 and it wasn’t going to work. I already knew I was wasting my time asking the hosting company to upgrade the version; I had tried that before with another shell program that was obsolete and they (politely) declined4. So, time to get programming!

I already know that the python version on the hosting is ancient and doesn’t have SSL support at all5, ditto Perl, so I am going to have to use PHP. I won’t have a problem with that as there are versions up to 7.1 on this host and the SSL libraries are properly current too. Command-line PHP isn’t great, but it will work just fine for this application.

Here is the resulting simple program to send a notification using the Pushbullet API. This is called from a Bash script if the WordPress verification checks fail; the resulting warning message turns up on all my registered devices.

<?php
/**
* PHP program to send messages to pushbullet.
* Usage: php message-pushbullet.php creds='...' title='...' body='...' [quiet] [headers]
*/
date_default_timezone_set('UTC');
error_reporting (E_ALL);

$url = "https://api.pushbullet.com/v2/pushes";
$type = "note";

if(php_sapi_name() == 'cli'){
    //convert cli params to GET params (chop off name of program which is 1st)
    parse_str(implode('&', array_slice($argv, 1)), $_GET);
}
//process parameters
$credentials = isset($_GET['creds']) ? trim($_GET['creds']) : "";
$title = isset($_GET['title']) ? trim($_GET['title']) : "";
$body = isset($_GET['body']) ? trim($_GET['body']) : "";
$quiet = isset($_GET['quiet']) ? true : false; //silent mode
$headers = isset($_GET['headers']) ? true : false; //show headers for debugging

if(!$credentials || !$title || !$body){
    echo "Usage: message-pushbullet creds='...' title='...' body='...' [quiet] [headers]\n";
    exit(1);
}

$post_fields = json_encode(compact('type', 'title', 'body'));

$ch = curl_init();

$options = [
    CURLOPT_URL => $url,
    CURLOPT_TIMEOUT => 30,
    CURLOPT_MAXREDIRS => 6,
    CURLOPT_HTTPHEADER => ['Access-Token: ' . $credentials, 
                            'Content-Type: application/json',
                            'Content-Length: ' . strlen($post_fields)],
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => $post_fields,
    CURLOPT_HEADER => $headers,
];
curl_setopt_array($ch, $options);

$result = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);

if(!$quiet){
    echo $result;
}
//check ok
if($status == 200){
    exit (0);
}
//failed
exit(1);

/* end */

Command-line parameter handling isn’t great as standard, so I cheat and turn them into GET parameters6. This handily means the program would work if used as a web script too with no further effort.

Oddly the API docs specify the parameters as JSON, but the version Ashley Rich created uses standard POST-style variables set using the CURL -d option and this works too. The Pushbullet API talks about maintaining backwards-compatibility to earlier versions, so I suspect there was a pre-JSON version at one time.

Most of the time was spent sorting out the Curl parameters needed 🙂

 

Simple Python SMTP mail script – Part 2

Part 1 explained just why I wanted to send a SMTP email from the command line and explained some of the constraints I was under; this is how I got there.

I started out by not really being that familiar with Python, but I will always contend that knowing how to program is independent of the choice of language, so I wasn’t worried.

Fortunately (?), the very old version of Python on my shared hosting meant that I didn’t need to concern myself with the Python 2 vs Python 3 debate7, or which library to choose for some functions (no choice) so I could just get on with it.

Since I wanted to make this a general-purpose tool, I spent some extra time getting SSL on ports 465 and 587 working as well as non-encrypted on port 25. My laptop has the latest version of Python 2.7, so I could test these even if I couldn’t use them on TSOHost.

Here is the script:

#!/usr/bin/env python
#
# Simple smtp mailer program. Designed to work on old versions of Python (2.4).
# Has lots of command line parameters, but intended to run from shell scripts
# so doesn't matter. Tested on ports 25, 465 (encrypted) and 587 (encrypted).
# Only supports plain text auth via username & password.
# The encrypted modes require 2.6 or later as smtplib.SMTP_SSL is not present
# in earlier versions.
#
import smtplib
import datetime, sys, getopt, re
#import email.utils not on tsohost
#from email.mime.text import MIMEText not on tsohost
#SMTP_SSL not in tohost so port 465 and 587 won't work

port = ''
host = ''
mfrom = ''
mto = ''
msubj = ''
username = ''
password = ''
force_encrypt = False
quiet = False

def usage(progName = ''):
    print("Usage: %s -p port -o host -f mailFrom -t mailTo -s subject [opts] 'messageBody'" \
          % progName)
    print("Other options:")
    print("-u username -w password : Where login is required")
    print("-e : Force encryption - prevent plaintext username/password if not encypted")
    print("-q : Quiet")
    print("-h : Help")
    print("messageBody will be taken from stdin if parameter not present")
    
#get command line options    
try:
    myopts, args = getopt.getopt(sys.argv[1:],"p:o:f:t:s:u:w:eqh")
except getopt.error,e:
    print(str(e))
    usage(sys.argv[0])
    sys.exit(2)
 
for o, a in myopts:
    if o == '-p':
        port = a
    elif o == '-o':
        host = a
    elif o == '-f':
        mfrom = a
    elif o == '-t':
        mto = a
    elif o == '-s':
        msubj = a
    elif o == '-u':
        username = a
    elif o == '-w':
        password = a
    elif o == '-e':
        force_encrypt = True
    elif o == '-q':
        quiet = True
    elif o == '-h':
        usage()
        sys.exit()
#check for minimal required args
if(port == '' or host == '' or mfrom == '' or mto == '' or msubj == ''):
    print("Missing args:")
    usage(sys.argv[0])
    sys.exit(2)
        
#read from stdin for message body if no further args
if(len(args) == 0):
    mbody = sys.stdin.read()
else:
    mbody = args[0]

#generate correct mail date format    
date = datetime.datetime.now().strftime("%a, %d %b %Y %H:%M:%S%z")
#generate message header and body
msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" \
       % (mfrom, mto, msubj, date, mbody)

#set to 1 for lots of debug messages
debuglevel = 0
#indicate encrypted
encrypted = False

try:
    if(port == '465'):
        smtp = smtplib.SMTP_SSL(host, port)
        encrypted = True
    else:
        smtp = smtplib.SMTP(host, port)
    smtp.set_debuglevel(debuglevel)
    smtp.ehlo()    
    #print(smtp.ehlo_resp)
    if(re.search(r'^STARTTLS\b', smtp.ehlo_resp, re.M | re.I)):
        #print("STARTTLS accepted")
        smtp.starttls()
        smtp.ehlo()
        encrypted = True
        #print(smtp.ehlo_resp)
    #see if login required
    #(order of login & plain not specified and there can be other options)
    if(re.search(r'^AUTH\b.*(?:LOGIN\b.*PLAIN\b|PLAIN\b.*LOGIN\b)', \
                 smtp.ehlo_resp, re.M | re.I)):
        #print("Plain login accepted")
        if(force_encrypt and not encrypted):
            print("Error: Plain text login over non-encrypted connection not allowed.")
            smtp.quit()
            sys.exit(2)
        else:
            smtp.login(username, password)
    #send the mail!
    smtp.sendmail(mfrom, mto, msg)
    smtp.quit()
    if(not quiet):
        print("Mail sent successfully.")
except Exception, e:
    print("Error: unable to send mail.")
    print(str(e))
    sys.exit(2)
#end

That’s all there is to it. The response from smtp.ehlo() is tested to see if STARTTLS is supported; if it is then encryption is kicked in before doing a plain text login provided AUTH PLAIN LOGIN is supported. If the port is 465 then the sequence is started in SSL mode instead by using smtplib.SMTP_SSL.

Usage looks like:

./send-smtp-mail.py -p 25 -o smtp.mail.host \
-f "A User <a.user@me.co.uk>" \
-t "me@me.co.uk" \
-s "Py Test Mail" \
"Test Message Here..."

Pretty simple really and I was pleased with how well the result worked. Python didn’t take much brainpower to get the basics going, at least at the noddy level needed to make something like this work.

It helped to be familiar with the workings of mailservers though. This is the legacy of getting my own home mailserver working using Dovecot & Postfix and having to get relaying through a port 465 tunnel to Virgin Media’s servers working when Postfix doesn’t support port 4658.

Simple Python SMTP mail script – Part 1

TSOHostI host this blog via TSOHost in the UK. On the surface they look like just another low-cost hosting provider, but under the covers they are actually quite programmer-friendly. My cheapo hosting plan has SSH access, cron jobs, up-to-date PHP7 and Mysql and surprisingly good tech support. All good so far and way in excess of the average WordPress user’s needs.

However, I’m not an average (or sensible) WordPress user, so when I started actually writing blog posts I also started thinking about off-site backups and it was far too easy and boring just to install a backup plugin and call it done. As a minor part of my day job I am involved in using Amazon web services, mostly S3 for cloud storage so I started musing about sticking my blog backups in a S3 bucket.

I already had a bash script to create a db backup, tar the wp-content dir, gzip it all up and send it to S3, so stick it on a cron job and pretty much job done eh? … Not quite; I like to get a daily email reminder that the backup ran and some info about the result.

…and that is when things started to unravel…

Shared hosting is cheap, but it comes with quite a few limitations compared to non-shared alternatives9. The main problem is that shared hosting is a magnet for every spammer and scammer around, so the hosting providers have to severely limit access to normal utilities like command-line mail and sendmail. Even when these utilities are there and work, the outgoing mail host is probably on every blacklist around so your mail is going nowhere. This is true for TSOHost too; I couldn’t use any of the easy ways of sending mail from my backup script.

However, all is not lost, they provide an internal SMTP server inside their cloud server farm that can send mails and I already know that works fine as I have used it from one of my PHP applications. Problem solved?

Not quite. Another issue with shared hosting appears – a lot of expected utilities are missing from the command line (ssmtp? – nope) and what is there is old. Like – really old10. They can’t be updated and nothing can be installed except user-land programs.

So how about doing something is a scripting language? TSOHost runs the latest PHP and I could pull down PHPMailer and whip a script up pretty quickly; I’m pretty good with PHP, so that would be easy. It just feels messy though – I started with a single bash script and I’d end up bolting another script and external libraries on the side just to send an email.

How about Perl? Version 5.8.811 is installed and it has the Net::SMTP module. It doesn’t have the Net::SMTP::SSL module though, but that is ok as the TSOHost internal SMTP server doesn’t need encrypted connections. Yes?

No. I don’t really know Perl beyond doing superficial one-liners in shell scripts; plus there is something …painful about Perl12. I’ll consider that as a last resort.

How about Python? Version 2.4.313 is there, the smtplib module loads. Again, SSL won’t work as that didn’t come in until v2.6, but not a problem here. Yes?

Yes! I don’t know Python well either, but that’s what Skynet-Beta Google is for eh?

To be continued…

 

WordPress Plugin WP-Mail-SMTP Self-signed Certificate Patch

I am using an excellent plugin by Callum Macdonald to send mail from my WordPress blog. This works absolutely fine with the various mail-servers that I have pointed it at, but embarrassingly not with my own home mail-server14.

Now, this isn’t exactly a problem since I don’t need to send mail via my own home mail-server from my blog, but as with all things geeky I got curious why it wouldn’t work. I know the mail-server works fine15 and I can connect and send /receive from all my computers and devices, so I was a bit puzzled. What was especially odd is that it worked from WAMP on my laptop, but not from a test site in a Vagrant virtual machine on the same laptop. So I went Googling and found this. Doh! My WAMP installation is still on PHP 5.516, but the Vagrant instance is on 7.1 – and I still have a self-signed certificate on my mail-server.

So; just because I could17, I created a very simple WordPress plugin to apply a filter to WP Mail SMTP to add the extra parameters for self-signed certificate connections. The code looks like this:

add_filter( 'wp_mail_smtp_custom_options' , function( $phpmailer ){

    $phpmailer->SMTPOptions = array(
        'ssl' => array(
            'verify_peer' => false,
            'verify_peer_name' => false,
            'allow_self_signed' => true
        )
    );
    return $phpmailer;
});

Couldn’t be simpler eh? This can be stuck in functions.php or in a plugin like I did18.

My plugin can be downloaded here: WP-Mail-SMTP Self-signed Certificate Patch Plugin.

Now; just a work of caution – you really shouldn’t do this. You should replace your self-signed certificate with a real one from Let’s Encrypt or a commercial provider. There’s not much excuse for using self-signed certificates now that it is so easy and cheap / free to get a real one (except for testing).

I had even less excuse as I already have a Let’s Encrypt certificate on Apache running on the same physical server as the mail-server19. After making and testing this plugin I actually fixed the certificate problem properly and I can confirm that exactly the same certificates that work for Apache (and Nginx) also work with Postfix. The only remaining issue that I need to sort is to modify the routine that re-starts Apache on certificate renewal to also restart Postfix.

A few minutes work and I’ve got until June to do it 😉