As a die hard Unix guy and ex Slashdot-esque zealot (colloquially a bit of a twat). Yet I'm knocking out PowerShell all the time now and dread having to log into the pile of CentOS kit I have lying around. It is arcane. Even after 15 years I spend most of my time in the manpages or working out another damn config format.
Literally did a two liner to scrape a web page, parse it and call a REST endpoint with the parsed data as JSON in PowerShell. I cant even use python now. I've been broken.
It makes me cringe saying all this as well kind of like a racist making amends with his past.
I'm not even a Powershell fan, I'm just not especially fond of Unix-likes, which feel like a local maximum that has turned into an unchangable standard. I think Unix-like environments feel dated, clunky and there's a tendancy for it to result in permanent bloat, stagnation and a web of interdependencies.
I understand that lots of people are familiar with Unix-like tools and their many idiosyncrasies and like to be able to transfer that knowledge across platforms. But that doesn't mean there isn't merit to a platform not conforming to that and trying to improve on it.
Lots of people keep using the word "bloat" to refer to Linux as opposed to Windows, which I find bizarre given that it's long been a criticism in the other direction.
Since the word is used mainly as an insult due to people having emotional attachments to technology, the real question in my mind is how did you not find it bizarre on every use.
Or I guess another way to say it is that "bloated" is only a measurement of the speaker's disdain for the software under question. It has little relationship with any actual quality measurement, generally. Bizarre on its face, to me.
Man after years of hearing how bloated Windows is compared to other OSes, I was expecting my first Windows Phone to be the same. But nope to my surprise it was fast and fluid, even on low end devices! I guess, the same can also be said about Win8/Win10 compared to Vista/Win7.
Bloat seems like dirt that accumulates overtime. A little bit of cleanup and optimization can sometimes do magic.
Bloat and legacy support are two sides of the same coin.
Microsoft chose one answer to that, Apple another.
I think Windows Phone is a clear example that Microsoft has technical chops when starting clean, it just lacks business willpower to make decisions that favor performance over support.
Yeah, I would genuinely like to know what all is considered to be bloat within the Unix space. If anyone could point to some information about this I'd be curious.
Not only that but they include more documentation inside the binary. Someone also pointed out to me one time that some features in some GNU utils are only possible to have if they are implemented in the binary, rather as a separate program.
Well, generally speaking nothing is impossible, but it's often more efficient to put the extra features in the same binary. However this "bloat" is nothing compared to your typical desktop or programming environment.
I disagree.
1. Unix-like environments are (or rather were before systemd) kinda similar and once you drop a shell - good to go. Even with powershell Microsoft have gone down the route of version/feature pinning. If you plan to maintain at least Windows 7/Server 2003 in your scripts - well, powershell 2 is the highest you can aim, bye bye new features, welcome bloat. Red Hat while shipping old versions still backports features.
2. The only thing in Unix-likes being standard are core utils and OpenSSH. Everything else can be anything (SysV/upstart/Systemd, Gnome/KDE/tiny_tiling_wm, bash/zsh) and we are actually deprecating older technologies for new ones (quite often changing the name). Due to ELTS (extremely long term support, often called legacy cruft) core subsystems are mostly the same (on the outside) in Windows. And if it is a server you can bet it is running IIS and MSSQL.
3. Try using `wmic` shell or `net` util. Unbearable clunkiness of such tools and bearable GUI tools mean that we have very little general scripts/tools to transfer
Don't know if it's any good --- never tried it; it says it's about half complete, but I don't know if it's a useful half.
After looking at the docs, you could do a lot of what Powershell does with Unix shells; you'd need a different set of conventions, where instead of using unformatted text as an intermediate format you used a streamable table format with support for metadata. Then you could have commands like 'where', which would be awesome.
...or something. sh syntax is a bit lacking; PowerShell's got lots of useful builtins, including having native support for the format so it knows how to present it to the user. An sh version would conversion routines back and forth from text.
The tricky part would be bootstrapping; getting enough functionality quickly enough that enough people would start using it to make it sustainable.
I'd still rather use this than faff around with awk, though. I've done way too much of that. And if I never have to parse the output of ls -l using cut again, I will be a happy person.
> Don't know if it's any good --- never tried it; it says it's about half complete, but I don't know if it's a useful half.
There are at least two parties working on it who have interest in certain features to be working: It's the NuGet shell within MonoDevelop, so custom PowerShell hosts need to work as well as a bunch of things needed by NuGet. Then one of the more prolific developers actually gets paid to work on Pash to support features related to PowerShell add-ins and a few others. (My own efforts so far were mostly bits and pieces, missing cmdlets (there are still a lot of those), test cases, and weird parser behaviour, mostly due to my history of golfing in PowerShell – golfed code makes for some fun tests of edge cases.)
Isn't this just complaining you want a different tool? The entire Powershell format depends on, amongst other things, the object interface being sane, existing, and usable.
The whole point of the GNU system was working around the usual "I can see it here, need this bit, and want to put it there". If you need to do something really specific a lot, you write a tool which does that.
Exactly. The Unix/Linux world has not gone the way of a Powershell type shell for the simple reason that for the uses where we want an object-oriented API, we have a plethora of scripting languages designed for that purpose, and we all have our own favourites.
When we want a text oriented API we use shell scripts. When we want an object oriented API we pick our favourite scripting language - which may very well for many of us be different for different problem domains.
Yes, precisely. The main thing would be to do is to define the stream format, and then persuade people to actually use it.
The point of the exercise is that the traditional Unix pipelinable commands have standardised on unstructured textual data. Powershell has standardised on structured tabular data, which is what lets you do the cool things.
Yeah, same with the find -printf (at least macports can install coreutils and give you "gstat"). Portability is hard :-/ http://mywiki.wooledge.org/ParsingLs
and flavor the output elements and format to suit. (this sorts by date well,
but would do less well on file or symlink names with embedded newlines - swap
\0 for \n to wrestle with those.)
> Literally did a two liner to scrape a web page, parse it and call a REST endpoint with the parsed data as JSON in PowerShell.
This sounds cool. Can you recommend terse resources for getting to know PowerShell? Is MSDN the best place to look for docs or are there better places to go?
TBH I couldn't find a decent book or resource and sort of hacked my way around for a year or so. It had a good build in manual (get-help). This is a good poke at the fundamentals of my example:
Scrape a page:
$flight = " LH3396"
$url = "http://bing.com?q=flight status for $flight"
$result = Invoke-WebRequest $url
$elements = $result.AllElements | Where Class -eq "ans" | Select -First 1 -ExpandProperty innerText
Hit a REST endpoint:
$body = @{
Name = "So long and thanks for all the fish"
}
Invoke-RestMethod -Method Post -Uri "$resource\new" -Body (ConvertTo-Json $body) -Header @{"X-ApiKey"=$apiKey}
This is really cool (and unfortunately I know nothing about powershell, I'm young and dumb) but I'm pretty sure this is replicable line for line with bash (and curl, maybe awk too idk). Am I wrong?
The main difference between bash/gnuutils stuff and Powershell is that every command returns an object (or an enumeration of objects), and commands can take an object (or an enumeration of objects) as input.
This lets everything implicitly understand how to access named properties without everyone having to do string parsing.
Sure, you can solve the same problems with bash/grep/awk/sed/etc - but sometimes it's a bunch simpler to solve in Powershell.
wow, didn't even know of jq but look's super cool. was trying to avoid using packages you would have to 'install' since it seemed that all the functionality the post I responded to used built-in's for powershell commands.
Probably not. In my experience it's not so much about things being impossible elsewhere, just that PowerShell can often be better at fiddling in a REPL until you got the results you wanted, in a way. The point where you need (or want) to upgrade to a more powerful language is IMHO earlier in bash than in PowerShell².
PowerShell handles objects like Unix utilities handle text. You get a lot more orthogonality in commands, e.g. there's just a few commands dealing with JSON, CSV, XML, etc.¹ – they mostly do just the conversion of a specific text-based format to an object list or tree representation and back. Everything else after that can be done with just the same set of core commands which deal with objects. Filtering, projecting, and iterating over object sequences is probably the most common part of PowerShell scripts and one-liners and it's a part that's useful everywhere you need the language. Note that we have a bit of that in the samples here, too. ConvertTo-JSON is used to convert a hashmap to a JSON string to use in a request, and Invoke-WebRequest already handles parsing HTML for us so the following line can just filter elements by certain attributes.
Where the ideal format for the core Unix tools you use most frequently, is free-form text, you often have special commands that work on other formats by replicating a few core tools' features on that format, e.g. a grep for JSON, a grep for XML, etc. There are others, of course, that do the conversion in a way that's friendly to text-based tools, but the representation still lacks fidelity. Finding XML elements with certain attributes quickly becomes an exercise in how to write robust regexes. Cutting up CSV by column numbers is a frequent occurrence – and since the tools do not understand the format it makes for not a pretty read in the script's code. Personal opinion here, based on lots of PowerShell written, some shell scripts written, lots of horrible things read in either (sure, awful PowerShell scripts do exist, but I'd argue that discovering a nice way is much easier where you can cut most of the ad-hoc parsers written in regex or string manipulation in pipelines).
(One last point about orthogonality of commands: ls has a few options on how to sort or output the results, for example. Sorting a list of things? That's Sort-Object's domain. Formatting a list of things? Format-Table, Format-Wide, Format-List. That's quite a bit less each individual command has to do and it's all just for nicety to the user. For working with the output programmatically you don't (well, and can't) need them at all.)
And finally, let's not forget that PowerShell exists on Windows where text-based tools are mostly useless. Want to query the event log? The registry? WMI? Good luck. Windows has a long history of having non-text formats everywhere in the system and for administration it's a bit hard to pretend they don't exist. Jeffrey Snover elaborates a bit on that here: http://stackoverflow.com/a/573861/73070.
There may be a lot of developers getting by with Unix tools on Windows, but they're not the target audience for PowerShell³. And the previous approach to scripting things on Windows servers and domains was either batch files or VBScript/JScript.
This ... uhm, got longer than anticipated and probably a lot less coherent than planned. Apologies for anything that makes no sense. I didn't have caffeine yet.
______
¹ ConvertFrom-JSON and ConvertTo-JSON for JSON for example. For CSV there are also convenience commands that directly work on files as well, so there's four of them. XML processing is built-in since .NET can do that easily already.
² I probably don't get to live the rest of the day now, I guess.
³ I am a developer, though, who uses PowerShell daily for scripting, as a shell, and a .NET playground. The PowerShell team at MS was a bit surprised once when I told them that my background was not server administration (the Scripting Games were quite focused on that part and often involved doing things with WMI or AD – stuff I rarely, if ever, do.)
Your comment was useful, so not totally in vein :-)
Just incited me to have a little look at powershell (read through [0], useful intro). It looks nice, I can definitely see the utility in have a simple object model for transferring information between processes. In nix land you get pretty good at extracting data from simple text forms, though sometimes it's harder than it should be.
One thing that jumped out at me there is the overhead of the commands.
430ms: ls | where {$_.Name -like "*.exe"}
140ms: ls *.exe
27ms : ls -Filter "*.exe".
Not so much the absolute numbers but the fact that there are 3 different ways of doing it and the more flexible choice is over a magnitude slower.
What happens when you add another command to the pipeline? Do they buffer the streams like in linux?
I guess the situation will improve over time but how complete is the eco-system at the moment? One area nixes will always shine is the total ubiquity. Everything can be done over commands and everything works with text.
You found three ways of doing things that all do filtering at a different level. The -Filter parameter employs filtering on the provider side¹ of PowerShell, i.e. in the part that queries the file system directly. Essentially your filter is probably passed directly to FindFirstFile/FindNextFile which means that fewer results have to travel fewer layers. The -Path parameter (implied in ls .exe as it's the first parameter) works within the cmdlet itself, which is also evident in that it supports a slightly different wildcard syntax (you can use character classes here, but not in -Filter) because it is already over in PowerShell land. The slowest option here pushes filtering yet another layer up, by using the pipeline, so you get two cmdlets, passing values to each other and that's some overhead as well, of course. Note that the most flexible option here is by combining Get-ChildItem with Where-Object, whereas the direct equivalent in Unix would probably be find which replicates some of ls' functionality² to do what it does, which probably places it in a very similar spot to ls, performance-wise.
It's not uncommon for the most flexible option to be the slowest, though. In my own tests my results were 18 ms, 115 ms and 140 ms for doing those commands in $Env:Windir\system32, so the difference wasn't as big as in your case. For a quick command on the command line I feel performance is adequate in either case, unless you're doing things with very large directories. If you handle a large volume of data, regardless of whether it's files, lines, or other objects, you probably want to filter as much as you can as close to the source as you can – generally speaking.
As for buffering ... I'm not aware of, unless the cmdlet needs to have complete output of the previous one to do its work. Every result from a pipeline is passed individually from one cmdlet to the next by default. Some cmdlets do* buffer, though, e.g. Get-Content has a -ReadCount parameter that controls buffering in the cmdlet (man gc -param readcount). Sort-Object and Group-Object are the most common (for me at least) that always need complete output of the stage before to return anything, for quite obvious reasons.
However, even though I did some work on Pash, the open-source reimplementation of PowerShell, I'm not terribly well-versed in its internal workings, so take the buffering part with a grain of salt.
As for completeness, well, the Unix ecosystem has an enormous edge here, simply by having been there for decades and amassing tools and utilities. Since PowerShell was intended for system administrators you can expect nearly everything needed there to have PowerShell-native support. This includes files, processes, services, event logs, active directory, and various other things I know little to nothing about. Get-Command -Verb Get gives you a list of things that are supported directly that way. It seems like even configuration things like network, disks and other such things are supported by now. At Microsoft there's a rule, I think, that every new configuration GUI in Windows Server has to be built on PowerShell. Which means, everything you can do in the GUI, you can do in PowerShell, and I think you can in some cases even access the script to do the changes you just made in the GUI – e.g. for doing the same change on a few hundred machines at once, or whatever.
Of course, you can just work with any collection of .NET objects by virtue of the common cmdlets working with objects (gcm -noun object). For me, whenever there is no native support, .NET is often a good escape hatch, that in many cases isn't terribly inconvenient to use. You also have more control over what exactly happens at that level, because you're one abstraction level lower. As a last resort, it's still a shell. It can run any program and get its output. Output from native programs is returned as string[], line by line, and in many cases that's not worse than with cmd or any Unix shell.
_____
¹ Keep in mind, the file system is just one provider and there are others, e.g. registry, cert store, functions, variables, aliases, environment variables that work with exactly the same commands. That's why ls is an alias for Get-ChildItem and there is no Get-File, because those commands are agnostic of the underlying provider.
² So much for do one thing – but understandable, because ls' output is not rich enough to filter for certain things further down in the pipeline.
Oh yeah, I can see how the different filtering spots would make a difference.
Was just a bit surprised at the overhead of adding a command to the pipeline. A similar setup on linux would be like the following I guess (on a folder with 4600 files, 170 matching files).
20ms time ls *.pdf
35ms time find -maxdepth 1 -iname '*.pdf'
60ms time ls | egrep '.pdf$'
I was more wondering if adding each additional command added so much overhead. Your numbers looks much more reasonable. Maybe the article I read had a big step due to filesystem / caching or something.
One thing that became clear with Heartbleed is people apply security updates and don't restart the necessary services to pick up the updates. Rebooting the entire OS is the nuclear option, but it's incredibly effective.
This is a Hyper-V hypervisor host on a private VLAN three layers behind the internet and two layers behind the users locked in a room with no console access other than via an ILO card on yet another VLAN.
There has only been one KB which infers a security problem and that has had alternative mitigation put in place.
Not rebooting your server doesn't imply incompetence or disregard to security.
It must be great to know no-one within your environment will ever do anything wrong. Or download anything that will do anything wrong. Or visit any websites than can hijack your browser into doing anything wrong.
Being 'private' only means you don't get hit by drive-by scans from the Internet. There are (depending on configuration) plenty of opportunities for internal attacks, for example the workstation being used to access the boxes. Not to mention removable media (usb, cd-rom) or files copied onto those otherwise isolated hosts could be infected.
Patching servers is just good practice. As is designing a system that can handle rebooting individual servers without user-facing downtime.
Power shell is ok but MS DOS terminal needs an update: it has several limitations about fonts, encoding and missing libs that make it the worst terminal experience imho.
Ah also POSIX compliance could simplify the life of those who develop on Windows. I tryed to develop on Windows for many years, I tryed hard but you are always missing some feature that on Unix is basic.
How can two lines of code be earning you shit tons of cash? Can you show it if you swapped out specific details with generic placeholders (which page you're parsing, which endpoint you're hitting)?
It merely exploits some fundamental differences on how humans operate between two markets and turns that into real-time information for me on which I can make trade decisions.
This isn't stocks, shares, futures etc for ref either.
I don't want to say any more nor post any code as it's unique and a good earner at the moment.
> It merely exploits some fundamental differences on how humans operate between two markets and turns that into real-time information for me on which I can make trade decisions.
> It merely exploits some fundamental differences on how humans operate between two markets and turns that into real-time information for me on which I can make trade decisions
Let me guess: arbitrage applied to sports gambling. Wink my way if I'm on the right track.
Sounds awesome, how would you recommend someone get started with PS? I'm CLI-only dev, use Node, code tons of front-end but all I use to do is CMDer, VIM, and SSH all running together.
As a die hard Unix guy and ex Slashdot-esque zealot (colloquially a bit of a twat). Yet I'm knocking out PowerShell all the time now and dread having to log into the pile of CentOS kit I have lying around. It is arcane. Even after 15 years I spend most of my time in the manpages or working out another damn config format.
Literally did a two liner to scrape a web page, parse it and call a REST endpoint with the parsed data as JSON in PowerShell. I cant even use python now. I've been broken.
It makes me cringe saying all this as well kind of like a racist making amends with his past.
The only thing I still hate is windows update.