Meeting will happen in #d9readiness on drupal.slack.com.
| Gábor Hojtsy (he/him) | Using rector to automate deprecated API use removal |
| Gábor Hojtsy (he/him) | @shaal had a good experience setting this up, using some steps that were not necessarily evident |
| Gábor Hojtsy (he/him) | do you want to write up a blog post about this @shaal? |
| shaal | it's probably going to be more a length of a tweet |
| Gábor Hojtsy (he/him) | @shaal hm, was there public docs you used that I missed? |
| Gábor Hojtsy (he/him) | re your ambitious proposal of core inclusion I don’t think that is possible at all due to heavy 3rd party dependencies |
| Gábor Hojtsy (he/him) | but it would be great to have a service where you press a button, it checks out the latest branch of the projects, runs this on it and gets you a patch? |
| Gábor Hojtsy (he/him) | it would be really painful to add this to d.o but a 3rd party service is not unimaginable if someone is sufficiently inspired to do |
| shaal | So yeah, my experience was that it was very easy to set up.And at this stage it's just a POC.It seems like the main part that is missing, is adding each deprecation (do we have a list of most popular deprecation to the least?)So this tool, at any stage, even if partially working, the deprecations that are already included will help everyone to get to D9 faster/better. |
| dan2k3k4 | rector/rector or drupal8-rector/drupal8-rector ? |
| shaal | I was testing https://github.com/drupal8-rector/drupal8-rector |
| shaal | @Gábor Hojtsy (he/him) do you want to change the title of the thread to Drupal8-Rector ? |
| mirom | I tested version from Pronovix people some time ago, was there any progress? The very first version wasn’t very usable. For D9 it might be good policy that if you deprecate something, you should add a rule for rector, so it can be removed easily. |
| shaal | yes! |
| shaal | I wish that upcoming code deprecations for D10, will also include the rector code, which will make D9->D10 much easier and smoother. |
| Gábor Hojtsy (he/him) | http://hojtsy.hu/blog/2019-may-24/analysis-top-uses-deprecated-code-drup... are the top deprecations |
| Gábor Hojtsy (he/him) | I opened an issue at https://github.com/drupal8-rector/drupal8-rector/issues/21 to cover those |
| Gábor Hojtsy (he/him) | (opened in May) |
| Gábor Hojtsy (he/him) | we need people to work on this, that is what it comes down to, @mxr576 gifted this to the community but is not working on it |
| mirom | can we use rector as is to upgrade between symfony versions given there already are rules for that? |
| Gábor Hojtsy (he/him) | do you want to try? |
| mirom | sure, my christmas are empty for now |
| shaal | Rector already have this for Symfony -Let's say you pick the symfony40 set and you want to upgrade your /src directory:# show a list of known changes in Symfony 4.0vendor/bin/rector process src --set symfony40 --dry-run# apply upgrades to your codevendor/bin/rector process src --set symfony40 |
| shaal | (from https://github.com/rectorphp/rector#a-prepared-sets) |
| shaal | And it can upgrade your code from PHP 5.3 to PHP 7.4 |
| shaal | The report from May says it found 7843 calls to the deprectated function drupal_set_messageThis function is already part of the existing POC, so in theory no one has to manually fix drupal_set_messagehttps://github.com/drupal8-rector/drupal8-rector/blob/master/src/Rector/... (edited) |
| dan2k3k4 | tested out drupal8-rector but I see it gives weird results like the following diff: |
| mxr576 | It automatically adds all use statements that could be necessary when it replaces deprecated code, because it cannot know whether those are already inherited from the parent or not. So you have to remove those with code style checkers later, or with PHPStorm's "Optimize imports" feature.Btw, there is an open PR that is probably related to this, that I haven't had time to finish yet: https://github.com/drupal8-rector/drupal8-rector/pull/15/files |
| mxr576 | we need people to work on this, that is what it comes down to, @mxr576 gifted this to the community but is not working on itYeah, I would like to continue the work on it, but I do not have a capacity for this in my free time nowadays. In working hours, I am mostly traveling and participating in workshops this month. I am trying to follow Rector related threads here. I am very happy that the community started to see the real potential in Rector and also ready to contribute to the project |
| Gábor Hojtsy (he/him) | Simpletest removal, discussion of base classes and contrib module (edited) |
| Gábor Hojtsy (he/him) | @lendude suggested #3091462: [policy, no patch] Agree to purge WebTestBase extending *TestBase classes outside the Simpletest module from D9 and |
| Gábor Hojtsy (he/him) | #3093621: [META] Plan to get a stable D8/D9 release of the SimpleTest module |
| lendude | So the main question about the base classes was, if we remove them from core, we might still end up breaking contrib tests, which we are trying to prevent with the whole move-to-contrib thing |
| lendude | If the answer is ‘yes we are fine with this small break’ then we can just close it and start to remove all those base classes |
| dww | True, but as catch writes in the issue, the un-breaking solution is a very small change to the use statement. And if there's an official release of simpletest contrib now/soonish that provides them, folks can already start using that and porting their tests to it. |
| dww | (if they don't want to already do the right thing and port to the newer base classes) |
| dww | FWIW, as someone holding down the "please don't break us" position a lot these days, I'm +1 on a break of this kind. ;) |
| lendude | Ah ok, so the confusion might be over which base classes I’m refering to, I mean like ViewsTestBase and BlockTestBase and the like, these will not be moved to contrib. |
| lendude | Or that was never part of the plan as far as I know |
| lendude | They will just get deleted |
| lendude | So if you are using them you will have to make your own version of them |
| lendude | The same goes for all the test traits that are not in the simpletest module in core but spread through all the other modules |
| dww | OHH, yeah, I totally misunderstood. Why are we removing ViewsTestBase? Is there an issue about that? |
| xjm | Those have mostly been deprecated for a very long time |
| xjm | ViewsTestBase extends WebTestBase and etc. |
| dww | Oh, sorry, I'm confused. ViewTestBase (not deprecated) vs. ViewsTestBase (deprecated). |
| lendude | Yeah, so we have two versions of all those base classes, a WebTestBase version and a BrowserTestBase version, same goes for all the test traits, and the WTB versions have been deprecated |
| lendude | I think they are both calles ViewsTestBase, just with a different namespace |
| dww | Doesn't look like it... see core/modules/views/tests/src/Functional/ViewTestBase.php |
| dww | No s in there. But yeah, kinda confusing. |
| lendude | Yeah but for block/comment/node they will probably be the same (views always has to be different ) |
| dww | So, we don't appear to have gotten any clarity in here. Do we need a plan? Do we just stop holding the hands of people still using WTB? |
| lendude | Well at the very least I’ll update the IS a bit to make it clearer what specific cases this is about |
| Gábor Hojtsy (he/him) | Disruptive markup changes, when are they possible? |
| Gábor Hojtsy (he/him) | @dww raised this |
| Gábor Hojtsy (he/him) | if @lauriii would be online, I would summon him |
| dww | E.g. #3078334: Datetime and Datelist element titles use inaccessible <h4> for field labels |
| dww | (Dang Slack... that's "Datetime and Datelist element titles use inaccessible
for field labels") |
| Gábor Hojtsy (he/him) | #2659890: [Plan] Drupal 9 markup and CSS backwards compatibility is the high level plan |
| dww | andrewmacpherson wants us to use a fieldset for datetime, not our weird hacks. That'd be a Big Change(tm). It's mostly working, but I don't really want to sink too much time into it if the answer is "we can make changes of that scale in D10" |
| dww | Re: 2659890 -- Indeed. See my comment #37. I specifically asked about this and there's been no answer. I know we're all juggling Too Many Things(tm), so no judgement/shame. I'm just looking for answers so I can plan my time/efforts accordingly. Thanks! |
| xjm | D9's Stable will be the latest markup output by modules |
| xjm | The internal themes and the module markup can change in minors; stable will change with each major |
| xjm | However, we're also retaining D8's Stable in core for BC |
| xjm | So it's like the PHP deprecation process in that way -- the public API only changes in the major version and only based on changes already made with BC |
| dww | Righto. So between now and stable_9 being "tagged" / "forked" etc, we change module markup, can fix internal themes, but leave classy and stable broken. And folks extending / using stable or classy remain broken until they port their stuff. Is that a fair summary? |
| xjm | "broken"? |
| xjm | I don't understand what you mean |
| dww | Inaccessible. #states doesn't work, etc. ;) Depends on the issue in question. |
| xjm | We've made changes to Stable and Classy before for accessibility fixes |
| dww | To fix these bugs, we have to change markup. |
| dww | But we can't change markup or we break BC. |
| xjm | It's one of the categories of disruptive bug fixes we do change in the stable base themes |
| dww | Okay, but other bugs don't have accessibility implications. |
| xjm | Depending on the impact vs. the disruption |
| dww | E.g. #states, #description_display not being honored, etc. |
| xjm | I can't speak to those specific issues... up to the frontend framework manager |
| xjm | But those sound like render API and not templates |
| dww | Well, it's a case of the core templates not handling stuff render API says should be available. |
| xjm | Anyway, yes, the correct approach is still to fix what can be fixed with BC in minor releases. The major release version is being started basically now, same as with backend deprecations ending with the 8.8 beta |
| dww | What about a situation where a render API change means an entire template and preprocess() pipeline is dead code? E.g. we stop having a separate datetime-wrapper* and we just re-use fieldset ? |
| xjm | Really should be discussed on the specific issue, but the continuous upgrade path always means we should be making the change with BC in minor releases first |
| Gábor Hojtsy (he/him) | Change policies for Drupal 9 about deprecation vs. new API addition timing? |
| Gábor Hojtsy (he/him) | @dww proposedmy recent experience makes me wish we had a 1 minor version delay between introducing the good new alternative of something and marking the old way deprecated... things marked for deprecation in 8.8.x that already have existing solutions in 8.7.x are easy. When we add the alternative and immediately deprecate in the same minor version, contrib maintainers have to add technical debt to our modules to still support old and new approaches conditionally...(edited) |
| Gábor Hojtsy (he/him) | we can still improve the 8.8 deprecation messages to add notes where the code was available before |
| Gábor Hojtsy (he/him) | the 8.7 ones go into full-on possibility in a month, so not sure that makes any difference anymore, but the 8.8 ones would be good to revisit in terms of the deprecation text to be more clear when the replacement API was already there before |
| larowlan | the alternative is when we don’t deprecate - we end up with a long tail of technical debt because new usages sneak into core. If you contrast say the entity.manager split (an early deprecation which didn’t fail tests) with the later ones, it took us muuuuucccch longer to clean up the debt |
| larowlan | but it would be ideal if we could get smarter about the errors |
| larowlan | almost like a wrapper around trigger_error that used some version parsing smarts |
| mikelutz | Realistically, attempting to postpone the deprecation 1 minor version isn’t practical for a number of reasons. It’s technical debt that just won’t get fixed, and not adding the deprecations immediately means that we wouldn’t catch new core patches that might use old code we are about to deprecate. |
| berdir | Yes, not deprecating results in 2 API's and it's hard to use the right one. See current assert methods that we undeprecated where method autocomplete in phpstorm is not very helpful |
| Gábor Hojtsy (he/him) | another alternative for Drupal 9 to 10 is stop deprecating two minor releases out |
| mikelutz | but @larowlan is right, the answer is in tooling and the new deprecation message format. |
| mikelutz | Because we need to not say deprecation suppression is on or off in the drupalci.yml. We need to declare the deprecation level that the contrib project expects to be compatible with. |
| Gábor Hojtsy (he/him) | I’m trying to find examples of deprecation messages where we added a note of prior replacement, cannot find them, but I know I’ve seen them. |
| mikelutz | We need to say deprecation-level: 8.7 and then suppress all deprecations added in 8.8+ during that test run. (edited) |
| Gábor Hojtsy (he/him) | @mikelutz that would still require branching for the two versions to support core support timelines |
| Gábor Hojtsy (he/him) | (and be able to do security fixes) |
| berdir | And for things that we are deprecating properly, I really don't see the problem. we had a long discussion on this already and I can't say anything that I've said before: getting deprecation messages in contrib is a good thing. It means that your module still works, and you can actually rely on core ensuring this with legacy test coverage. And yes, better tools there will help, by if you can say I require 8.6 and then you don't want to see deprecations for 8.6 and older. |
| mikelutz | @Gábor Hojtsy (he/him) no more so than waiting a minor to add the deprecation. In my case, contrib would declare their deprecation level even with the security minor or the stable minor, whichever they prefer. |
| mikelutz | We would still ADD the deprecation at the time we add the new api, but contrib wouldn’t SEE it until they chose, be that 1 minor later, or 2 minors later, or right away. |
| mixologic | This is something I've had opinions on for a long time. Deprecation messages should be treated like phpcs or other 'standards' violations and not treated as a build failure, unless the maintainer wants that. |
| berdir | where we need to be extra careful is things that break contrib, even if it's just tests and make that as easy as possible to deal with, or things were we can easily be forward compatible. E.g. backporting just the interface for TrustedCallback to 8.7 would allow contrib to use that that alreeady right now without ugly hacks that have to be cleaned up again. |
| mixologic | The issue people keep running into is "how to I have 'passing' tests" - which wouldnt necessarily be the target if people could have varying degrees of control over what a deprecation means in the context of phpunit. |
| mikelutz | At the end of the day, this is all optics. No matter what you do you still have the same 3 options. 1) wait to fix deprecations until your supported core has the replacement, 2) use funky hacks to get a version that works with all core versions 3) make a new branch for a new core minor and up the minimum supported core version and fix your deprecations there. |
| mixologic | But the phpunit symfony bridge seems to either give us useless counts if we dont want to fail the build. What we need is the full error messages, but not fail tests. |
| mikelutz | @mixologic precisely. Your module is not broken or incompatible with a newer core versions because tests fail due to deprecations. |
| Gábor Hojtsy (he/him) | @mikelutz @mixologic filtering deprecation messages does not solve the problem of not being able to be N and N+1 major version compatible with the same codebase due to the overlapping security coverage of N.last-1 and N+1 while N.last-1 has deprecated APIs (edited) |
| mikelutz | But it does look bad, and I see why maintainers want to keep green test runs. |
| mixologic | right. They shouldnt fail. Unless you want them to. Testing against the security/supported versions, yeah, fail on deprecations. Testing against the prerelease or dev version.. well, no deprecations shouldnt be a failure. |
| mixologic | @Gábor Hojtsy (he/him) you mean in the instance where N.last-1 only has the deprecated api, N.last has both the deprecated API and the new API, and N+1 only has the New api? |
| dww | Yeah, my specific proposal is probably won't fix. It's tech debt for core. I'm just a bit sad that core avoiding technical debt means contrib has to add our own, or we have to stop supporting things we should keep supporting. |
| mikelutz | @Gábor Hojtsy (he/him) I don’t understand why it doesn’t. Yes, it you want to support security, stable, and development then you are limited to the api level of the security version, but your same codebase is still compatible with stable and development minors too. Your tests aren’t failing, they are just giving deprecation messages that you can’t fix yet because you have chosen to remain on the security version api. |
| mixologic | I think thats also a somewhat confusing stance that core took with regards to the 'security support' of core, and adding in that additional 6 months. It was always somewhat ill defined whether or not contrib maintainers should support that version of core etc |
| dww | It's also too bad d.o prevents a 9.x-* branch if that's the "recommended" solution to avoid technical debt. If I need a new branch to work with 9.x, let me tag it as such. Yeah, I know we hope that contrib will be Pure Semver(tm) sometime Soon, but that's not working yet. So meanwhile, it's confusing and weird that I'm supposed to make "8.x-2.0" as my D9-compatibile release. (retracted) (edited) |
| mixologic | Well, its not that we're preventing a 9.x-* branch its that they're no longer called 9.x-* |
| mikelutz | If we allowed it, it would break the plan to add semver. |
| mixologic | but yeah, we cannot have a situation where there exists the same version on both 8.x and 9.x |
| mikelutz | I know it’s a bit weird, but when we drop the 8.x/9.x it would be bad if you have 8.x-2.0 and 9.x-1.0. |
| dww | Fair enough. |
| mixologic | If we were to get a little bit more finesse we could allow 9.x- for the time being, and just not allow releases to be made for versions that already exist as 8.x- versions. |
| mixologic | But that would also be confusing and weird. |
| dww | Right, the time is better spent marching towards the future, not helping the past limp along behind us. |
| mikelutz | Right, if you did 9.x-3.0 you would be fine, there’s just no easy way to communicate or force that. |
| mixologic | So, this is sortof a dumb idea, but what if we had a different deprecation strategy. |
| mixologic | Right now the strategy is, say method_A() is old |
| mixologic | gets marked deprecated and becomes method_B(). |
| mixologic | Then, when drupal 9 comes out, 'deprecated code is removed' |
| mixologic | what if |
| mixologic | nevermind. IM going to suggest what we're already doing. |
| mikelutz | lol. |
| mikelutz | I thought you were going to say we should just leave poor method_A() alone. I mean, what’s wrong with him anyway, and why is method_B() any better.. |
| mixologic | I was going to say leave it there in 9.0, and just have it point to B, until 9.1 and then remove it |
| mixologic | It's almost as if we should just not do deprecation removal at the major version boundary only, and instead deprecate in version N, targeted for removal in N+3 |
| mixologic | or N+4 or something |
| mikelutz | that’s bad semver. |
| mixologic | that is true |
| Gábor Hojtsy (he/him) | @mikelutz the problem with deprecations in N.last-1 is that it stops being core-supported when N+1 comes out |
| mixologic | Yeah, I think the real answer here is that 8.7 should have been the final place for deprecations for 9.x, and 8.8 and beyond should have been all 10x deps |
| Gábor Hojtsy (he/him) | IMG_20191111_205643.jpg |
| mikelutz | Okay, you are talking about major version boundaries. (edited) |
| mixologic | but I think that is only the benfit of hindsight |
| Gábor Hojtsy (he/him) | so if your contrib wants to support N and N+1 majors then it can not do that before N+1 comes out, it can only do that with some delay |
| Gábor Hojtsy (he/him) | yeah that is what I wrote in https://drupal.slack.com/archives/CDDD98AMN/p1573501439231900?thread_ts=... |
| mikelutz | Yes, fine. And that’s true. |
| Gábor Hojtsy (he/him) | unless, we stop deprecating in N.last-2 |
| mixologic | Right, and not 'stop deprecating' but 'stop deprecating for removal in N+1', they become 'deprecated for removal in next major +2' |
| mikelutz | I think that’s too much. |
| mikelutz | But I also think that contrib branching isn’t a bad thing. They shouldn’t have to support that security branch other than security fixes. |
| mikelutz | If someone is on the security core version, they probably don’t want active development on their contrib either. |
| mixologic | So, we need tooling that can warn/distinguish between "removed in the next major version" or "removed in the next major version +1" |
| mixologic | we already need that now, as we're kinda hamstrung in deprecating things in the 8.9/9.0 cycle |
| Gábor Hojtsy (he/him) | @mikelutz branching is a difficult psychological thing, previously because people were forced to do a new branch for Drupal 6, 7, 8, etc. they took the opportunity to refactor their stuff even if the API diff was not that significant (eg. between 6 and 7) — if we can decouple branching from supporting a new major version, then the “let’s refactor this shit now that I have a new branch” would not happen at the same time in contrib as the new core major is coming out |
| Gábor Hojtsy (he/him) | its not just a tooling question, it is a human question as well |
| mikelutz | Sure, but it’s a psychological thing, and its why we need contrib semver. once it’s a minor branch and not a major branch the psycological burden will be lowered. |
| dww | Note: even though Git is Great, I also consider a new branch "technical debt" I have to maintain. It would still require more effort on my part to maintain my module, I have to remember to cherry-pick stuff at the right time, create separate releases + release nodes, etc, etc. |
| mikelutz | but you don’t. It’s not a supported branch, it’s just the last version to work with the old core. |
| dww | Not if it's my 8.7.x-compatible branch. |
| dww | And I have bugs (inconceivable, I know!), and I need to fix them. |
| mikelutz | but not in your 8.6 branch. |
| mikelutz | And not in your 8.7 branch after December 4. |
| dww | Huh? my 8.7.x branch needs to live until 8.9.0 is out. |
| Gábor Hojtsy (he/him) | @mikelutz why? if there is a security release, Drupal 8 is still supported until November 2021… even if a maintainer stops fixing regular bugs |
| mikelutz | If core is not fixing bugs in 8.7 after December 4th, why do you feel you need to fix bugs in your contrib branch for 8.7? |
| Gábor Hojtsy (he/him) | @mikelutz core will fix at least security bugs in 8.7 until next summer (edited) |
| mikelutz | Yes, and the possibility of needing to do a security backport would be the purpose of contrib creating a new branch. But what are the odds of any particular module needing to do that, and how huge is the actual added workload in the event that that happens? |
| mikelutz | If you chose not to backport bugfixes to that branch. |
| berdir | and if we consider long-term, then I assume that following the removal of the 8.x prefix, we'll also allow contrib to do patch releases? (because right now each release is exposed to composer as a minor release). So instead of having to start a new major version, you can just do a new minor version, and a security/bugfix release could still be a patch release |
| dww | Yes, the plan is to allow major.minor.patch(extra) |
| dww | (just like we already do with core) |
| dww | whether contrib maintainers will use the right values for these based on what they're doing is anyone's guess. ;) |
| dww | (in fairness, just like now) |
| berdir | and until then, I believe I've already seen a workaround that goes like this: you have 8.x-1.0, then you release 8.x-1.1 that requires 8.7, now you have to do a security release and want to still support 8.6, then you could do 8.x-1.2 based on 8.x-1.0, document that this is based on 8.x-1.0 and only has the security fix, and at the same time release 8.x-1.3 that is based on 8.x-1.1 and has the security fix too but isn't flagged as a security release (so that people could update to 1.2 and don't get a security-updates message). not perfect, but I could live with that until we have full semver (edited) |
| Gábor Hojtsy (he/him) | @berdir that sounds like a recipe for someone to bork a site who tries to update and gets downgraded |
| berdir | if they update from 1.1 to 1.2, yeah, but you could document that in the SA. anyone who just does updates the module with composer/drush update would get 1.3, you need to specifically download 1.2 for it to break. as I said, it's not great, but I'd prefer that over doing a new major version until we have patch releases |
| berdir | also, you could likely even mark 1.2 as incompatible with D8.7, which means it would be impossible to update from 1.1 to 1.2 as 1.1 would require 8.7. So I think this would work quite well. just like composer will be smart enough to only update to 1.2 then as long as you're still on 8.6 |
| Gábor Hojtsy (he/him) | ok I posted the blog post I had drawn up the figures for in my head last week in our discussion with @dww https://twitter.com/gaborhojtsy/status/1194010636763705344 |
| Alex Moreno | thanks for that post/graph @Gábor Hojtsy (he/him), I know it's going to be pretty useful for some conversations ^^ |
| Alex Moreno | and everyone else |