Problem with scrollrect dragging

I'm newbie to Unity GUI. I make a scrollview with gridlayout as content. Add some items under the grid. However, no matter how many items I add and how far I drag, when I release dragging the items, they all reset to the origin positions. Shouldn't they scroll to some new places I drag? Anyone facing with the same problem?

Version 5.4.0.