I've also used Netdata for years on many servers now. One of the problems with it is its way to fail silently when some healtcheck or alarm is configured badly instead of erroring out on the process start. I understand the reasoning behind it - it tries to monitor by default as much as possible and if some collector fails for whatever reason (let's say you don't have MySQL installed) then it will be just disabled and only way to know that is by looking at the dashboard and not finding it there. I'm only using local agent and not the cloud which means that if server dies totally then there's also no monitoring insights - something to keep in mind.
Creating complex software is not at all complicated, and sometimes it is done right at the beginning of a project, even before solving a single business problem.
But this is also kind of a problem created by Google - why force from 30 to 33 and not force from 30 to 31, then 31 to 32 etc? Gradual API level upgrades would make more sense in my world and probably would be less risky for every party.
I'm the OP and thank you for thinking along with me here. As stated in numerous replies already then I totally agree that I could have done better in terms of testing things out - of course, there's always room for improvement in that regard. There was a deadline set by Google (again, first time I heard about it was at 18th of August, not before), change seemed trivial at time and since app worked on an old Android version as it was before then I didn't expect it to fail so miserably. Again, I'm not a seasoned Android dev, but have 15+ years of experience in software development in general so I have some expectations how things will work or not and what to expect and be afraid of. I really didn't know that "best practice" is to do a staged roll-out of "99.99999999%" to have a way of partial "yank" possibility of the latest release. To find out that there's no way to cancel/delete a latest release to fall back to previous working version was just something I did not expect in my wildest dreams (I guess this is something you only learn during situations like these). Yes, everyone can blame me for not testing every functionality with every Android version and I do the same, but please open your eyes and understand that the way releases are currently handled by Play Store is not a sane person would do outside of Play Store. Everyone will have a problem like this at one point and I do hope that this article will and thread in here will lower the number of developers experiencing situation similar to this.
I'm the OP and wanted to clarify in case you missed some points - it is a legacy application which does not have any active dev teams on it and needs only developers attention when Google says so and as mentioned by multiple other commenters here the first time I got that e-mail from Google, was at 18th of August. I would not agree that I have been lazy, but instead trying to solve this problem in the time-constraints set by Google and failing to do so because of the inability to put a fix to production and/or pull back current release version. Of course I admit that there's always ways to improve quality assurance.
There has been zero communication towards me from Google until two weeks until deadline. Yes, maybe if I would have logged into Play Console then there might have been some notifications, but there have been no reason to do so until that e-mail (I'm usually not involved with Android projects, otherwise I might have noticed similar warnings via other projects early on).
Double-checked - no e-mails prior 18th of August from "Google Play Console". Also, double-checked "Google Play Console" (even under "Archived messages") - first message about deprecation I see is from 17th of August (a day before e-mail). Even if there were something a year ago, then 3 month, 1 month followup would make sense instead of 2-3 week reminder.
Maybe it's possible that different roles in "Google Play Console" will receive different type of messages at different time? I'm not an admin for that app and it's possible that someone else has gotten prior warnings, but might have ignored these, because usually they are not with technical background.
Actually nowadays it's not that bad anymore. Android browser itself offers installable PWA and there is an event called beforeinstallprompt event (https://developer.mozilla.org/en-US/docs/Web/API/Window/befo...), which can be used to perform PWA installation on user interaction. Of course it's not supported in every browser.
iOS is more difficult since user needs to understand that "saving to home screen" is same as installing "app" and there's no way to trigger it programmatically or help user in any other way than with visual illustrations.
The share menu is used for everything on ios, I had to get used to it at my first apple device’s case.
One notably stupid usage of the Share option was (I believe it is no longer how it’s done) adding an image to a hidden folder — that’s something you definitely don’t want to share, yet quite easy to accidentally send to someone during this process.
Thankfully this insanity is changing in the upcoming release due in September. A button for a 3 dot/hamburger menu appears in the bottom right when you select photos.
Because it doesn't make sense that it's part of the web share functionality to begin with.
The native share dialog is about "I have this piece of content, now share it with one of these other apps", e.g. saving a document locally, or sending it with AirDrop, or saving to Google Drive, or sending as an email attachment, etc.
Installing a site as an app on your homescreen has absolutely nothing to do with sharing content to begin with.
Typical "legacy application situation where no team is assigned to" here. At least I've never seen in my 15+ year career a legacy application, which has a very good maintenance/testing model in place. It will just not work because maintenance/testing needs also resources and updates, but if priorities are in different places then it's not possible. Of course as stated already multiple times - things could have handled in better ways, but this doesn't mean that Google should not allow to stop release at least. Anyway, lessons learned.
I agree that things could have been done better in many ways. However, as explained, it is a legacy application, which does not see any active development nowadays and it would just not make sense to build such a robust QA. This particular app have been written long time ago by another company and there's not even simple unit tests. That's the hard truth. But still, even as complex setup as you have, there's still going to happen mistakes and the real problem is that there is no way to pull-back/cancel/rollback release.
How would staged roll-out help in this situation for all customers? When end-user gets the faulty version of the app, does he/she have a way of getting the non-faulty version somehow?
Unfortunately, as you noted, the game is rigged and you have to play within the sandbox provided. With that, and as you also noted, your testing simply needs to be better, especially since this is a customer-facing app and not an app that's solely used internally. As for dealing with the rollout when a serious bug is now in production, if you didn't roll out your app to 100% of the user base you could halt the roll out so it wouldn't affect any more customers. Then, while far from ideal, you could ask affected customers to uninstall the app and then reinstall it, and the newly installed version would reflect the previous version of your app.
A-ha - good tip about roll-out, uninstall & install. Was not aware of that possibility. But yeah, it's still a hack and there should be a better way - just let me delete/cancel latest release even if roll-out is 100% for whatever reasons.
About better testing - there's always room to improve testing, but no way it's going to happen with a legacy application where no active team is assigned. Only these irregular updates mainly forced by Google are done. Unfortunately.
building a business on any big vendor (google, apple, amazon, microsoft, etc) without at least making sure these risks are shared with the client seems like a fast way to run into these costly overtime problems.
Rigged seems a bit extreme. They had to make a few changes to keep things up to date. That they tripped over this small hurdle and went tumbling down the stairs doesnt change that it's a small hurdle.
Just to add, even if you do not change anything at all, you will spend many hours just getting the build environment up to snuff if it hasn't been touched in over a year. Gradle, the plugins (android, crashlytics, etc), the build.gradle DSL, dependencies ... its quick bit rot.
Android as an OS does not support rollbacks. It can only upgrade apps in monotonically increasing `versionCode`s. The simple reason is that client side data may have been upgraded by a newer release to a format that is now incompatible with the old version. Sqlite databases follow the same policy.
This is well-known to anyone who has been doing Android development for a while.
Totally understand that a roll-back is more complex, but it doesn't explain why pulling back/pausing/deleting/yanking a release is not possible either - customers who got the newest faulty release can uninstall & install to get the previous version back (no need to worry about incompatibilities here) and the ones who did not get the updated version yet can stay using the old one until a new fixed version will be released.
This is exactly what staged rollouts are for. The author ignored that best practice. You can halt any release as long as it is in progress. Once you mark it complete (defined as rolled out to 100%), you can’t roll it back.
Each track in Google Play (alpha, beta, canary, prod) can hold up to one release at a time. It would get really confusing if it allowed more than one. And with the other rollout safeguards provided decelopers, it’s quite possible and very easy to do exactly what you’re asking for.
It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.
"A bullet-proof automated testing solution" isn't needed for the type of problem you describe. Anything that logs in (including a human on a real device) would catch it even if it does nothing else.
Agreed about conflicting message. And even better - e-mail was pretty vague and had to Google a separate link (https://support.google.com/googleplay/android-developer/answ...) for more info. If I could go back in time then I would press that button, which asks for time until November, because as happened to us - it was not possible to go back to previous version in any way.
Yeah you just saved me; was working late last night to get all updated. But still some issues; and was worried about testing. So thank you. Sorry you didn't see it on timme
Yes, that's definitely one side of the problem and I'm not chasing too much backwards-compatibility. My biggest concern in this particular situation is that there is no way (with Android, at least) to pull-back/cancel/rollback release and everything is blocked behind Google's review process. Why isn't it just possible to "yank" problematic release and continue showing previous release as the latest version. That would solve most of the issues within context of this problem.
Rollbacks allow malicious actors to /simply-easily/ circumvent device security and user preference. To allow rollbacks is to /significantly/ increase the attack surface of a device.
What do you mean by that? Are you effectively trying to say that allowing upgrades does not have any risk of attack surface? I'm pretty sure that updating things have also a pretty high risk on introducing new previously non-existing security issues into your code-base/product.
Not necessarily, Google has access to the developer's private key they use for signing their APKs so they could just make a fake release that has a bigger version number than the current version whenever a rollback is needed. No change is needed on Android itself, it's a Google Play issue.
