Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#16645 closed defect (wontfix)

External dojox packages missing from git mirror

Reported by: Adam Peller Owned by: Eugene Lazutkin
Priority: undecided Milestone: tbd
Component: Operations Version: 1.8.3
Keywords: Cc: Colin Snover, ben hockey, cjolif, Eugene Lazutkin
Blocked By: Blocking:

Description

https://github.com/dojo/dojox

dojox/calendar is included in the main svn repository as an svn external. Perhaps it can be linked into the git repository the same way?

As an aside, for external repositories, it might be good to have something in the project which references the location of the linked repository so people can go there for more info or to contribute directly.

Change History (33)

comment:1 Changed 7 years ago by Adam Peller

Cc: Colin Snover ben hockey cjolif Eugene Lazutkin added

comment:2 Changed 7 years ago by cjolif

I think none of the external svn links is included. The same is true for dojox/app. This is not calendar specific. So if we do it for one, we must be consistent and do it for all.

comment:3 Changed 7 years ago by Adam Peller

Summary: dojox/calendar missing from git mirrorExternal dojox packages missing from git mirror

Agreed. Changing title to reflect this.

comment:4 Changed 7 years ago by bill

#16698 is a duplicate of this ticket.

comment:5 Changed 7 years ago by Adam Peller

who is the right owner for this?

comment:6 Changed 7 years ago by cjolif

Someone who has write access to dojo/dojox? If I well remember we restricted that to very few people at some point as the repo is read-only. Maybe Colin has?

comment:7 Changed 7 years ago by bill

Owner: changed from rchady to Eugene Lazutkin
Status: newassigned

Well, Eugene is really in charge of the mirrors. Eugene, can you fix? Or maybe we intentionally don't include those links?

comment:8 Changed 7 years ago by ben hockey

svn externals and git submodules are different in their approach and I wouldn't be surprised if it's not possible. svn externals track the tip of the external and don't require an explicit commit to get the newest content from the external whereas submodules work by pointing to a specific commit in a repo and require an explicit commit in the "parent" repo in order to point to a different commit in a submodule. Eugene is probably able to give more specific info related to the tool he's using but I would expect that at best the tool would need to create extra commits in order to update the subrepos or at worst it's likely that mapping externals to submodules is not supported.

comment:9 Changed 7 years ago by Adam Peller

If it's any help, I suspect some of these externals may be pointing to git repos... perhaps we can take a shortcut and just use git references rather than going through svn?

comment:10 Changed 7 years ago by Colin Snover

that does not solve that git submodules are fixed to specific commits.

comment:11 Changed 7 years ago by Eugene Lazutkin

Right now I use the simplest possible workflow: git svn fetch and git push. Last time I checked git svn didn't support svn externals. But keep in mind that the github mirror predates dojox/calendar, meaning that the use of svn externals without checking first was unwise and created the problem in the first place. I still don't understand why it was done in this fashion, instead of putting code in dojox like all other projects.

Obviously we can solve it. Providing that dojox/calendar is the only external project, we can extract it separately (is it svn?), copy to an appropriate place, and commit the whole tree. Obviously this simple solution doesn't scale up, and requires manual interventions if new projects are added in this fashion, so if we are to continue with this practice, we should have a more comprehensive solution for the future.

comment:12 Changed 7 years ago by cjolif

Actually the project that set that new practice is dojox/app. Followed dojox/calendar and dojox/dgauges. So there are 3 of them in dojox.

comment:13 Changed 7 years ago by Adam Peller

Eugene, I think this was done specifically to "help" with the 2.0/git migration, and because there's a moratorium on new dojox projects in svn. New projects like dojox/calendar should be in git (somewhere... it's nearly impossible for us to tell, nevermind the user) @cjolif: this too needs to be documented, perhaps in the README?)

If it's not too much trouble, it would be great to apply a fix to the latest 1.8 release as well as those going forward.

Last edited 7 years ago by Adam Peller (previous) (diff)

comment:14 Changed 7 years ago by Eugene Lazutkin

If they are already in github, why duplicate? Just for a convenience of having a common tarball? Are we going to do that with Dojo 2.0 too?

If we have a list of sources for external projects, I can add them to the main source tree.

More involved solution is to grab the svn checkout and parse it on the fly for all externals, converting github svn references to equivalent git references, and so on. Obviously it makes sense only if we are going to use this practice for a long time. Do I need to spend any time on that?

The mini solution: give me a list of such things, and I'll plug them in manually for now.

comment:15 Changed 7 years ago by Adam Peller

If they are already in github, why duplicate? Just for a convenience of having a common tarball? Are we going to do that with Dojo 2.0 too?

Yes, this is so they would be included in the 1.x (tarball) distribution, which is made via svn. In 2.0, svn goes away and we will use some github repo which I guess could use submodules to reference these same git respositories? Or perhaps we'll just rely on the package manager and use these git repositories directly?

The list is the three projects cjolif provided above:

  • dojox/app
  • dojox/calendar
  • dojox/dgauges

Thanks for handling this, Eugene

Last edited 7 years ago by Adam Peller (previous) (diff)

comment:16 Changed 7 years ago by ben hockey

as Colin and I have explained, git submodules are fixed to specific commits. what is the proposed strategy for moving that reference forward so that the submodules point to sensible commits? how will the history of the git repos be affected?

comment:17 Changed 7 years ago by ben hockey

also, is it too much for projects using the dojox git repo to have these other repos as siblings to dojox? I believe that consuming them in this fashion is more in line with our permanent move to git.

comment:18 in reply to:  17 Changed 7 years ago by Adam Peller

Replying to neonstalwart:

also, is it too much for projects using the dojox git repo to have these other repos as siblings to dojox? I believe that consuming them in this fashion is more in line with our permanent move to git.

Only as a last resort. To have a project set up a repository for development which looks different than the same release of Dojo on someone else's machine (which happened to be fetched from a tarball) is terribly awkward. Perhaps we never should have allowed these svn externals, but it's too late for that now.

comment:19 in reply to:  16 ; Changed 7 years ago by bill

Replying to neonstalwart:

git submodules are fixed to specific commits. what is the proposed strategy for moving that reference forward so that the submodules point to sensible commits?

Presumably something like:

$ git submodule update
$ git commit -a "point to latest versions of git packages"

To be done at the same time as the git svn fetch and git push.

comment:20 Changed 7 years ago by Eugene Lazutkin

I need to experiment first to ensure that the update will be smooth, then roll it out. It'll take 2-3 days for me.

comment:21 in reply to:  19 Changed 7 years ago by ben hockey

Replying to bill:

To be done at the same time as the git svn fetch and git push.

I'm not sure of the exact frequency of Eugene's script but it seems to be happening about once an hour. I don't think we want an extra commit per hour.

comment:22 Changed 7 years ago by bill

There's one shadow commit to git for every change in SVN, and there would be at most one commit for every change to the repositories like dojox/Calendar. What's the difference?

comment:23 Changed 7 years ago by Eugene Lazutkin

We cannot just copy files and commit, if there is a difference --- every extra commit changes the overall history and breaks git svn. I think we should shelve this ticket for now, and instruct users where to download extra modules. For official releases we can prepare a tar ball manually, or with a shell script.

comment:24 Changed 7 years ago by Colin Snover

Resolution: wontfix
Status: assignedclosed

comment:25 Changed 7 years ago by Eugene Lazutkin

I actually constructed something that works for now, but it may break with DojoX changes in SVN. Let's wait and see. If it breaks, I'll rollback my change and recreate our history.

comment:26 in reply to:  25 ; Changed 7 years ago by ben hockey

Replying to elazutkin:

I actually constructed something that works for now, but it may break with DojoX changes in SVN. Let's wait and see. If it breaks, I'll rollback my change and recreate our history.

will recreating history generate new hashes in the git repo for any of the commits? i feel very strongly that doing that is a really bad idea. people with submodules checked out will be broken if the commit they are referencing disappears out from under them.

comment:27 in reply to:  26 Changed 7 years ago by Eugene Lazutkin

Replying to neonstalwart:

will recreating history generate new hashes in the git repo for any of the commits? i feel very strongly that doing that is a really bad idea. people with submodules checked out will be broken if the commit they are referencing disappears out from under them.

Yes, if we are to re-push the history, last commits will likely to have new hashes. The old commits (up to the new history boundary) will be the same.

I am waiting for a DojoX commit. I suspect that any commit will show if it works or not, but I don't want to push a complete garbage. So the window of possible changes would be from about 2 hours ago to the very first commit.

comment:28 Changed 7 years ago by Colin Snover

Please, do not do anything that would require history to be rewritten on the main dojo repo. Test somewhere else first.

comment:29 in reply to:  28 Changed 7 years ago by Eugene Lazutkin

Replying to csnover:

Please, do not do anything that would require history to be rewritten on the main dojo repo. Test somewhere else first.

Thank you for your friendly guidance. Per your suggestion I removed the code to sync external repositories. Let's not touch it again lest something can be broken.

comment:30 Changed 7 years ago by ben hockey

eugene, could you please force push the repo back to the way it was before your last 3 commits - ie make 6b217c39a6c3148139f9c34038bfe4970456ccb1 the head

comment:31 in reply to:  30 Changed 7 years ago by Eugene Lazutkin

Replying to neonstalwart:

eugene, could you please force push the repo back to the way it was before your last 3 commits - ie make 6b217c39a6c3148139f9c34038bfe4970456ccb1 the head

That was done when I removed the code (see above).

comment:32 Changed 7 years ago by ben hockey

thanks - you almost gave me a heart attack :)

comment:33 Changed 6 years ago by bill

This is now fixed due to the move to github.

Note: See TracTickets for help on using tickets.