https://pulumi.com logo
Powered by
b

bright-motherboard-93404

09/21/2018, 5:56 PM
Hi! For some reason the 
user_data
that I'm supplying to my ec2 instances isn't getting run. When I log in with ssh and run the same command (python -m SimpleHTTPServer), everything works as expected. Is there any way to debug this situation?
this is the code (copied mostly from the python ec2 webserver example)
Copy code
group = ec2.SecurityGroup('web-secgrp',
    description='Enable HTTP access',
    ingress=[
       { 'protocol': 'tcp', 'from_port': 22, 'to_port': 22, 'cidr_blocks': ['0.0.0.0/0'] },
       { 'protocol': 'tcp', 'from_port': 8888, 'to_port': 8888, 'cidr_blocks': ['0.0.0.0/0'] }
    ])

user_data = """
#!/bin/bash
echo "Hello, World!" > index.html
nohup python -m SimpleHTTPServer 8888 &
"""

server = ec2.Instance('web-server-www',
    instance_type=instance_type,
    key_name=key_pair.key_name,
    security_groups=[group.name],
    user_data=user_data,
    ami=ami)
the 
key_pair
is a local .pem key that I use to be able to log in
m

microscopic-florist-22719

09/21/2018, 5:59 PM
@incalculable-sundown-82514 any ideas?
i

incalculable-sundown-82514

09/21/2018, 6:00 PM
can you look at 
/var/log/cloud-init-output.log
on your VM? If the user_data script ran, there should be some logs there.
If not, it’s possible we dropped it when creating the resource
b

bright-motherboard-93404

09/21/2018, 6:01 PM
thanks @incalculable-sundown-82514 I'll take a look. In what situations would we 'drop it' ?
i

incalculable-sundown-82514

09/21/2018, 6:01 PM
If there’s a bug in Pulumi, I mean.
b

bright-motherboard-93404

09/21/2018, 6:01 PM
aaah
haha, okay. Let me check the log and see
Yep, something went wrong there:
Copy code
Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~16.04.2 running 'modules:final' at Fri, 21 Sep 2018 18:01:08 +0000. Up 14.37 seconds.
2018-09-21 18:01:09,080 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [-]
2018-09-21 18:01:09,088 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2018-09-21 18:01:09,088 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 18.3-9-g2e62cb8a-0ubuntu1~16.04.2 finished at Fri, 21 Sep 2018 18:01:09 +0000. Datasource DataSourceEc2Local.  Up 14.49 seconds
@incalculable-sundown-82514 what would the above suggest? Is there something that's missing on the target machine? Wrong version of cloudinit in my local virtualenv?
i

incalculable-sundown-82514

09/21/2018, 6:20 PM
hmm, I’m not totally sure.
There’s not a lot of detail in this log about why these things failed 😞
b

bright-motherboard-93404

09/21/2018, 6:21 PM
Yes indeed
at least it could have told us the line or something
i

incalculable-sundown-82514

09/21/2018, 6:22 PM
can you see the contents of 
/var/lib/cloud/instance/scripts/scripts-user
?
maybe you could run that and see what happens?
b

bright-motherboard-93404

09/21/2018, 6:23 PM
Copy code
2018-09-21 18:01:09,088 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
2018-09-21 18:01:09,088 - util.py[DEBUG]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 806, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 188, in run
    return (True, results)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 76, in lock
    yield self._acquire(name, freq)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py", line 45, in handle
    util.runparts(runparts_path)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 850, in runparts
    % (len(failed), len(attempted)))
that's from 
cloud-init.log
Copy code
root@ip-172-31-41-50:~# cat /var/lib/cloud/instance/scripts/scripts-user
cat: /var/lib/cloud/instance/scripts/scripts-user: No such file or directory
i

incalculable-sundown-82514

09/21/2018, 6:23 PM
hmm
can you run 
pulumi stack export
on your stack and look for a json field named “user_data”?
just to confirm that Pulumi correctly passed user_data to AWS
b

bright-motherboard-93404

09/21/2018, 6:28 PM
what json key am I looking for? 
user_data
?
i

incalculable-sundown-82514

09/21/2018, 6:29 PM
yep, that’s the one!
Pulumi should have recorded it
pulumi stack export
basically dumps the entire state of your stack
b

bright-motherboard-93404

09/21/2018, 6:29 PM
Copy code
"userData": "\n#!/bin/bash\necho \"Hello, World!\" \u003e index.html\nnohup python -m SimpleHTTPServer 8888 \u003e /tmp/server 2\u003e\u00261 \u0026\nsleep 60\n"
it's there, with some minor changes that I just made (sleep 60)
i

incalculable-sundown-82514

09/21/2018, 6:29 PM
okay, thanks! So it would seem that we’re sending it to Amazon OK, but it’s failing to run for some reason
though the unicode escapes in there look a /little/ suspect
hmm
b

bright-motherboard-93404

09/21/2018, 6:31 PM
yes they do
oh oh
let me specify the user_data with u"blah"
maybe this is a python2/python3 thing here
i

incalculable-sundown-82514

09/21/2018, 6:32 PM
very possible
though if this is a unicode problem, it’s still a bug in Pulumi - we shouldn’t be messing up strings this badly
Python strings… 😢
b

bright-motherboard-93404

09/21/2018, 6:33 PM
Yeah, let's see if this is it. Would be good.
Still the same, even with user_data = u"thing"
same escapes
i

incalculable-sundown-82514

09/21/2018, 6:36 PM
hmm
and you’re using python 2?
b

bright-motherboard-93404

09/21/2018, 6:36 PM
I'm using python2, yes
i

incalculable-sundown-82514

09/21/2018, 6:39 PM
OK, what OS are you using?
b

bright-motherboard-93404

09/21/2018, 6:42 PM
Linux
Void linux to be precise -- locally, of course. The AMI is ubuntu
i

incalculable-sundown-82514

09/21/2018, 6:45 PM
okay, thanks. I’ll try to repro this
b

bright-motherboard-93404

09/21/2018, 6:46 PM
Thank you @incalculable-sundown-82514
i

incalculable-sundown-82514

09/21/2018, 6:50 PM
OK, so on my machine, this is what gets sent to AWS for user_data:
Copy code
"userData": "\n#!/bin/bash\necho \"Hello, World!\" > index.html\nnohup python -m SimpleHTTPServer 80 &\n"
this is on a mac using python 2.7.15
so the mystery is how these unicode escapes are appearing in the string
are those bytes in the source file by any chance?
b

bright-motherboard-93404

09/21/2018, 6:52 PM
Shouldn't be. Let me open it with another editor
no, but the string is a triple-quoted string. Maybe single-quotes might change this?
i

incalculable-sundown-82514

09/21/2018, 6:53 PM
It’s triple-quoted in my example, too
b

bright-motherboard-93404

09/21/2018, 6:54 PM
okay, well at least we have something to go on. I'll try to see why this keeps getting escaped
i

incalculable-sundown-82514

09/21/2018, 6:55 PM
ok, thanks! let me know if I can help at all
b

bright-motherboard-93404

09/21/2018, 6:57 PM
Will do Sean. I'm going to see whether this is a json encoder issue, perhaps. Try to wipe my virtualenv and only use pulumi
Well, this is strange -- the escapes are there is the export, but the userdata was run successfully. Well, at least I can see that it's some kind of weird interaction on my machine.
i

incalculable-sundown-82514

09/21/2018, 7:07 PM
that is very strange!
b

bright-motherboard-93404

09/21/2018, 7:14 PM
You know what, I think I can trace this back to a failed pulumi stack new, when it tried to install something outside a virtualenv
so I nuked the virtualenv and reinstalled things in order and now it all works
Thank you for your help Sean!
i

incalculable-sundown-82514

09/21/2018, 7:15 PM
yay! no problem, glad it worked out!
b

bright-motherboard-93404

09/21/2018, 7:15 PM
the unicode stuff is a red herring, probably an artifact of json encoding
So it worked once, and how this:
Copy code
2018-09-21 19:20:37,586 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'echo ok > ok'...'
i

incalculable-sundown-82514

09/21/2018, 7:39 PM
Now that is truly strange
b

bright-motherboard-93404

09/21/2018, 7:42 PM
this is with my old code, not the python webserver code, so the scenario is more complex. I set up userdata with the output of previous instances, so that workers can connect to a cluster leader.
"old code" == code that triggered the bug initially before I tried to repro on reduced example (python -m webserver)
i

incalculable-sundown-82514

09/21/2018, 8:43 PM
I am very surprised that the userdata is not 
text/plain
Powered by